bash test/kbcli/test_kbcli_0.9.sh --type 51 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-rkjff ` `kubectl create namespace ns-rkjff` namespace/ns-rkjff created create namespace ns-rkjff done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 123M 0 --:--:-- --:--:-- --:--:-- 123M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed set component name:minio set component version set component version:minio set service versions:0.9.0,2024.6.29 set service versions sorted:0.9.0,2024.6.29 no cluster version found REPORT_COUNT 0:0 set replicas first:2 set replicas third:2 set minimum cmpv service version set minimum cmpv service version replicas:2 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Error from server (NotFound): clusterdefinitions.apps.kubeblocks.io "minio" not found Not found topology in cluster definition minio LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-rkjff namespace. termination_policy:WipeOut create 2 replica WipeOut minio cluster check component definition set component definition by component version check cmpd by labels set component definition2: minio by component version:minio apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: minio-ocqhov namespace: ns-rkjff spec: terminationPolicy: WipeOut componentSpecs: - componentDef: minio name: minio replicas: 2 env: - name: MINIO_BUCKETS name: minio resources: limits: cpu: 100m memory: 0.5Gi requests: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_minio-ocqhov.yaml` cluster.apps.kubeblocks.io/minio-ocqhov created apply test_create_minio-ocqhov.yaml Success `rm -rf test_create_minio-ocqhov.yaml` check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Sep 01,2025 11:18 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:18 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:20 UTC+0800 check pod status done connect unsupported engine type: minio minio `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check pod minio-ocqhov-minio-0 container_name minio exist password CI89nk94g56V09eN check pod minio-ocqhov-minio-0 container_name lorry exist password CI89nk94g56V09eN No container logs contain secret password. describe cluster `kbcli cluster describe minio-ocqhov --namespace ns-rkjff ` Name: minio-ocqhov Created Time: Sep 01,2025 11:18 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-rkjff Running WipeOut Endpoints: COMPONENT MODE INTERNAL EXTERNAL minio ReadWrite minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9001 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME minio minio-ocqhov-minio-0 readwrite Running 0 aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:18 UTC+0800 minio minio-ocqhov-minio-1 readwrite Running 0 aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:20 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS minio false 100m / 100m 512Mi / 512Mi data:1Gi default Images: COMPONENT TYPE IMAGE minio docker.io/apecloud/minio:RELEASE.2024-06-29T01-20-47Z Show cluster events: kbcli cluster list-events -n ns-rkjff minio-ocqhov `kbcli cluster label minio-ocqhov app.kubernetes.io/instance- --namespace ns-rkjff ` label "app.kubernetes.io/instance" not found. `kbcli cluster label minio-ocqhov app.kubernetes.io/instance=minio-ocqhov --namespace ns-rkjff ` `kbcli cluster label minio-ocqhov --list --namespace ns-rkjff ` NAME NAMESPACE LABELS minio-ocqhov ns-rkjff app.kubernetes.io/instance=minio-ocqhov label cluster app.kubernetes.io/instance=minio-ocqhov Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=minio-ocqhov --namespace ns-rkjff ` `kbcli cluster label minio-ocqhov --list --namespace ns-rkjff ` NAME NAMESPACE LABELS minio-ocqhov ns-rkjff app.kubernetes.io/instance=minio-ocqhov case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label minio-ocqhov case.name=kbcli.test2 --overwrite --namespace ns-rkjff ` `kbcli cluster label minio-ocqhov --list --namespace ns-rkjff ` NAME NAMESPACE LABELS minio-ocqhov ns-rkjff app.kubernetes.io/instance=minio-ocqhov case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label minio-ocqhov case.name- --namespace ns-rkjff ` `kbcli cluster label minio-ocqhov --list --namespace ns-rkjff ` NAME NAMESPACE LABELS minio-ocqhov ns-rkjff app.kubernetes.io/instance=minio-ocqhov delete cluster label case.name Success cluster connect insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-minio-ocqhov" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-minio-ocqhov --namespace ns-rkjff ` Error from server (NotFound): pods "test-db-client-executionloop-minio-ocqhov" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-minio-ocqhov" not found `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-minio-ocqhov namespace: ns-rkjff spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "minio-ocqhov-minio.ns-rkjff.svc.cluster.local" - "--user" - "root" - "--password" - "CI89nk94g56V09eN" - "--port" - "9000" - "--dbtype" - "minio" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-minio-ocqhov.yaml` pod/test-db-client-executionloop-minio-ocqhov created apply test-db-client-executionloop-minio-ocqhov.yaml Success `rm -rf test-db-client-executionloop-minio-ocqhov.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 9s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 14s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 19s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 24s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 50s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 55s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 1/1 Running 0 60s check pod test-db-client-executionloop-minio-ocqhov status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-minio-ocqhov 0/1 Completed 0 66s check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Running Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:18 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:20 UTC+0800 check pod status done connect unsupported engine type: minio minio --host minio-ocqhov-minio.ns-rkjff.svc.cluster.local --user root --password CI89nk94g56V09eN --port 9000 --dbtype minio --test executionloop --duration 60 --interval 1 Execution loop start: Bucket executions-loop-bucket does not exist. Creating bucket... Bucket executions-loop-bucket created successfully. Execution loop start: executions_loop_1 [ 1s ] executions total: 33 successful: 33 failed: 0 disconnect: 0 [ 2s ] executions total: 78 successful: 78 failed: 0 disconnect: 0 [ 3s ] executions total: 124 successful: 124 failed: 0 disconnect: 0 [ 4s ] executions total: 170 successful: 170 failed: 0 disconnect: 0 [ 5s ] executions total: 215 successful: 215 failed: 0 disconnect: 0 [ 6s ] executions total: 261 successful: 261 failed: 0 disconnect: 0 [ 7s ] executions total: 303 successful: 303 failed: 0 disconnect: 0 [ 8s ] executions total: 346 successful: 346 failed: 0 disconnect: 0 [ 9s ] executions total: 388 successful: 388 failed: 0 disconnect: 0 [ 10s ] executions total: 434 successful: 434 failed: 0 disconnect: 0 [ 11s ] executions total: 477 successful: 477 failed: 0 disconnect: 0 [ 12s ] executions total: 517 successful: 517 failed: 0 disconnect: 0 [ 13s ] executions total: 558 successful: 558 failed: 0 disconnect: 0 [ 14s ] executions total: 598 successful: 598 failed: 0 disconnect: 0 [ 15s ] executions total: 638 successful: 638 failed: 0 disconnect: 0 [ 16s ] executions total: 681 successful: 681 failed: 0 disconnect: 0 [ 17s ] executions total: 728 successful: 728 failed: 0 disconnect: 0 [ 18s ] executions total: 776 successful: 776 failed: 0 disconnect: 0 [ 19s ] executions total: 821 successful: 821 failed: 0 disconnect: 0 [ 20s ] executions total: 866 successful: 866 failed: 0 disconnect: 0 [ 21s ] executions total: 908 successful: 908 failed: 0 disconnect: 0 [ 22s ] executions total: 950 successful: 950 failed: 0 disconnect: 0 [ 23s ] executions total: 992 successful: 992 failed: 0 disconnect: 0 [ 24s ] executions total: 1032 successful: 1032 failed: 0 disconnect: 0 [ 25s ] executions total: 1073 successful: 1073 failed: 0 disconnect: 0 [ 26s ] executions total: 1115 successful: 1115 failed: 0 disconnect: 0 [ 27s ] executions total: 1162 successful: 1162 failed: 0 disconnect: 0 [ 28s ] executions total: 1206 successful: 1206 failed: 0 disconnect: 0 [ 29s ] executions total: 1249 successful: 1249 failed: 0 disconnect: 0 [ 30s ] executions total: 1294 successful: 1294 failed: 0 disconnect: 0 [ 31s ] executions total: 1338 successful: 1338 failed: 0 disconnect: 0 [ 32s ] executions total: 1382 successful: 1382 failed: 0 disconnect: 0 [ 33s ] executions total: 1427 successful: 1427 failed: 0 disconnect: 0 [ 34s ] executions total: 1471 successful: 1471 failed: 0 disconnect: 0 [ 35s ] executions total: 1518 successful: 1518 failed: 0 disconnect: 0 [ 36s ] executions total: 1563 successful: 1563 failed: 0 disconnect: 0 [ 37s ] executions total: 1607 successful: 1607 failed: 0 disconnect: 0 [ 38s ] executions total: 1653 successful: 1653 failed: 0 disconnect: 0 [ 39s ] executions total: 1699 successful: 1699 failed: 0 disconnect: 0 [ 40s ] executions total: 1742 successful: 1742 failed: 0 disconnect: 0 [ 41s ] executions total: 1787 successful: 1787 failed: 0 disconnect: 0 [ 42s ] executions total: 1835 successful: 1835 failed: 0 disconnect: 0 [ 43s ] executions total: 1880 successful: 1880 failed: 0 disconnect: 0 [ 44s ] executions total: 1923 successful: 1923 failed: 0 disconnect: 0 [ 45s ] executions total: 1966 successful: 1966 failed: 0 disconnect: 0 [ 46s ] executions total: 2012 successful: 2012 failed: 0 disconnect: 0 [ 47s ] executions total: 2058 successful: 2058 failed: 0 disconnect: 0 [ 48s ] executions total: 2104 successful: 2104 failed: 0 disconnect: 0 [ 49s ] executions total: 2149 successful: 2149 failed: 0 disconnect: 0 [ 50s ] executions total: 2192 successful: 2192 failed: 0 disconnect: 0 [ 51s ] executions total: 2239 successful: 2239 failed: 0 disconnect: 0 [ 52s ] executions total: 2284 successful: 2284 failed: 0 disconnect: 0 [ 53s ] executions total: 2331 successful: 2331 failed: 0 disconnect: 0 [ 54s ] executions total: 2379 successful: 2379 failed: 0 disconnect: 0 [ 55s ] executions total: 2399 successful: 2399 failed: 0 disconnect: 0 [ 56s ] executions total: 2431 successful: 2431 failed: 0 disconnect: 0 [ 57s ] executions total: 2473 successful: 2473 failed: 0 disconnect: 0 [ 58s ] executions total: 2513 successful: 2513 failed: 0 disconnect: 0 [ 59s ] executions total: 2555 successful: 2555 failed: 0 disconnect: 0 [ 60s ] executions total: 2560 successful: 2560 failed: 0 disconnect: 0 Test Result: Total Executions: 2560 Successful Executions: 2560 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: minio Host: minio-ocqhov-minio.ns-rkjff.svc.cluster.local Port: 9000 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds Bucket: DB_CLIENT_BATCH_DATA_COUNT: 2560 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-minio-ocqhov --namespace ns-rkjff ` pod/test-db-client-executionloop-minio-ocqhov patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-minio-ocqhov" force deleted cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale minio-ocqhov --auto-approve --force=true --components minio --cpu 200m --memory 0.6Gi --namespace ns-rkjff ` OpsRequest minio-ocqhov-verticalscaling-t9bcc created successfully, you can view the progress: kbcli cluster describe-ops minio-ocqhov-verticalscaling-t9bcc -n ns-rkjff check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-verticalscaling-t9bcc ns-rkjff VerticalScaling minio-ocqhov minio Creating -/- Sep 01,2025 11:27 UTC+0800 check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:27 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:27 UTC+0800 check pod status done connect unsupported engine type: minio minio check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-verticalscaling-t9bcc ns-rkjff VerticalScaling minio-ocqhov minio Succeed 2/2 Sep 01,2025 11:27 UTC+0800 check ops status done ops_status:minio-ocqhov-verticalscaling-t9bcc ns-rkjff VerticalScaling minio-ocqhov minio Succeed 2/2 Sep 01,2025 11:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests minio-ocqhov-verticalscaling-t9bcc --namespace ns-rkjff ` opsrequest.apps.kubeblocks.io/minio-ocqhov-verticalscaling-t9bcc patched `kbcli cluster delete-ops --name minio-ocqhov-verticalscaling-t9bcc --force --auto-approve --namespace ns-rkjff ` OpsRequest minio-ocqhov-verticalscaling-t9bcc deleted `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check db_client batch data count `echo "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN;mc cat minioserver/executions-loop-bucket/executions_loop" | kubectl exec -it minio-ocqhov-minio-0 --namespace ns-rkjff -- bash` check db_client batch data Success test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-minio-ocqhov" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-minio-ocqhov --namespace ns-rkjff ` Error from server (NotFound): pods "test-db-client-connectionstress-minio-ocqhov" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-minio-ocqhov" not found `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-minio-ocqhov namespace: ns-rkjff spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "minio-ocqhov-minio.ns-rkjff.svc.cluster.local" - "--user" - "root" - "--password" - "CI89nk94g56V09eN" - "--port" - "9000" - "--database" - "" - "--dbtype" - "minio" - "--test" - "connectionstress" - "--connections" - "1024" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-minio-ocqhov.yaml` pod/test-db-client-connectionstress-minio-ocqhov created apply test-db-client-connectionstress-minio-ocqhov.yaml Success `rm -rf test-db-client-connectionstress-minio-ocqhov.yaml` check pod status check pod test-db-client-connectionstress-minio-ocqhov status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-minio-ocqhov 0/1 Completed 0 5s check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Running Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:27 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:27 UTC+0800 check pod status done connect unsupported engine type: minio minio 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket 2025-09-01T03:26:10.709Z, executions-loop-bucket Test Result: null Connection Information: Database Type: minio Host: minio-ocqhov-minio.ns-rkjff.svc.cluster.local Port: 9000 Database: Table: User: root Org: Access Mode: mysql Test Type: connectionstress Connection Count: 1024 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-minio-ocqhov --namespace ns-rkjff ` pod/test-db-client-connectionstress-minio-ocqhov patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-minio-ocqhov" force deleted check failover pod name failover pod name:minio-ocqhov-minio-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check db_client batch data count `echo "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN;mc cat minioserver/executions-loop-bucket/executions_loop" | kubectl exec -it minio-ocqhov-minio-0 --namespace ns-rkjff -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: benchtest-minio-ocqhov namespace: ns-rkjff spec: containers: - name: test-benchmark image: docker.io/apecloud/mc:RELEASE.2023-10-24T21-42-22Z command: ["/bin/sh"] args: ["-c", "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN && mc mb minioserver/bucket-okrwm"] restartPolicy: Never `kubectl apply -f benchtest-minio-ocqhov.yaml` pod/benchtest-minio-ocqhov created apply benchtest-minio-ocqhov.yaml Success check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-minio-ocqhov 0/1 ContainerCreating 0 0s check pod benchtest-minio-ocqhov status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-minio-ocqhov 0/1 Completed 0 5s `rm -rf benchtest-minio-ocqhov.yaml` `kubectl logs benchtest-minio-ocqhov --tail 30 --namespace ns-rkjff ` Added `minioserver` successfully. Bucket created successfully `minioserver/bucket-okrwm`. `kubectl delete pod benchtest-minio-ocqhov --force --namespace ns-rkjff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-minio-ocqhov" force deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in minio-ocqhov namespace. `kbcli cluster hscale minio-ocqhov --auto-approve --force=true --components minio --replicas 4 --namespace ns-rkjff ` OpsRequest minio-ocqhov-horizontalscaling-pkppj created successfully, you can view the progress: kbcli cluster describe-ops minio-ocqhov-horizontalscaling-pkppj -n ns-rkjff check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-horizontalscaling-pkppj ns-rkjff HorizontalScaling minio-ocqhov minio Creating -/- Sep 01,2025 11:29 UTC+0800 check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:27 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:27 UTC+0800 minio-ocqhov-minio-2 ns-rkjff minio-ocqhov minio Running notready 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:29 UTC+0800 minio-ocqhov-minio-3 ns-rkjff minio-ocqhov minio Running notready 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:31 UTC+0800 check pod status done connect unsupported engine type: minio minio No resources found in minio-ocqhov namespace. check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-horizontalscaling-pkppj ns-rkjff HorizontalScaling minio-ocqhov minio Succeed 2/2 Sep 01,2025 11:29 UTC+0800 check ops status done ops_status:minio-ocqhov-horizontalscaling-pkppj ns-rkjff HorizontalScaling minio-ocqhov minio Succeed 2/2 Sep 01,2025 11:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests minio-ocqhov-horizontalscaling-pkppj --namespace ns-rkjff ` opsrequest.apps.kubeblocks.io/minio-ocqhov-horizontalscaling-pkppj patched `kbcli cluster delete-ops --name minio-ocqhov-horizontalscaling-pkppj --force --auto-approve --namespace ns-rkjff ` OpsRequest minio-ocqhov-horizontalscaling-pkppj deleted `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check db_client batch data count `echo "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN;mc cat minioserver/executions-loop-bucket/executions_loop" | kubectl exec -it minio-ocqhov-minio-0 --namespace ns-rkjff -- bash` check db_client batch data retry times: 1 check db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart minio-ocqhov --auto-approve --force=true --namespace ns-rkjff ` OpsRequest minio-ocqhov-restart-ddkbg created successfully, you can view the progress: kbcli cluster describe-ops minio-ocqhov-restart-ddkbg -n ns-rkjff check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-restart-ddkbg ns-rkjff Restart minio-ocqhov minio Creating -/- Sep 01,2025 11:32 UTC+0800 check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running notready 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:34 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:33 UTC+0800 minio-ocqhov-minio-2 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:33 UTC+0800 minio-ocqhov-minio-3 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:32 UTC+0800 check pod status done connect unsupported engine type: minio minio check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-restart-ddkbg ns-rkjff Restart minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:32 UTC+0800 check ops status done ops_status:minio-ocqhov-restart-ddkbg ns-rkjff Restart minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:32 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests minio-ocqhov-restart-ddkbg --namespace ns-rkjff ` opsrequest.apps.kubeblocks.io/minio-ocqhov-restart-ddkbg patched `kbcli cluster delete-ops --name minio-ocqhov-restart-ddkbg --force --auto-approve --namespace ns-rkjff ` OpsRequest minio-ocqhov-restart-ddkbg deleted `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check db_client batch data count `echo "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN;mc cat minioserver/executions-loop-bucket/executions_loop" | kubectl exec -it minio-ocqhov-minio-0 --namespace ns-rkjff -- bash` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop minio-ocqhov --auto-approve --force=true --namespace ns-rkjff ` OpsRequest minio-ocqhov-stop-lwhvt created successfully, you can view the progress: kbcli cluster describe-ops minio-ocqhov-stop-lwhvt -n ns-rkjff check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-stop-lwhvt ns-rkjff Stop minio-ocqhov Creating -/- Sep 01,2025 11:35 UTC+0800 check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Stopped Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-stop-lwhvt ns-rkjff Stop minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:35 UTC+0800 check ops status done ops_status:minio-ocqhov-stop-lwhvt ns-rkjff Stop minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests minio-ocqhov-stop-lwhvt --namespace ns-rkjff ` opsrequest.apps.kubeblocks.io/minio-ocqhov-stop-lwhvt patched `kbcli cluster delete-ops --name minio-ocqhov-stop-lwhvt --force --auto-approve --namespace ns-rkjff ` OpsRequest minio-ocqhov-stop-lwhvt deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start minio-ocqhov --force=true --namespace ns-rkjff ` OpsRequest minio-ocqhov-start-sx4rh created successfully, you can view the progress: kbcli cluster describe-ops minio-ocqhov-start-sx4rh -n ns-rkjff check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-start-sx4rh ns-rkjff Start minio-ocqhov Creating -/- Sep 01,2025 11:35 UTC+0800 check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:35 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:36 UTC+0800 minio-ocqhov-minio-2 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:36 UTC+0800 minio-ocqhov-minio-3 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:37 UTC+0800 check pod status done connect unsupported engine type: minio minio check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-start-sx4rh ns-rkjff Start minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:35 UTC+0800 check ops status done ops_status:minio-ocqhov-start-sx4rh ns-rkjff Start minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests minio-ocqhov-start-sx4rh --namespace ns-rkjff ` opsrequest.apps.kubeblocks.io/minio-ocqhov-start-sx4rh patched `kbcli cluster delete-ops --name minio-ocqhov-start-sx4rh --force --auto-approve --namespace ns-rkjff ` OpsRequest minio-ocqhov-start-sx4rh deleted `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check db_client batch data count `echo "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN;mc cat minioserver/executions-loop-bucket/executions_loop" | kubectl exec -it minio-ocqhov-minio-0 --namespace ns-rkjff -- bash` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=minio-ocqhov,apps.kubeblocks.io/component-name=minio,apps.kubeblocks.io/vct-name=data --namespace ns-rkjff ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in minio-ocqhov namespace. `kbcli cluster volume-expand minio-ocqhov --auto-approve --force=true --components minio --volume-claim-templates data --storage 5Gi --namespace ns-rkjff ` OpsRequest minio-ocqhov-volumeexpansion-bftrm created successfully, you can view the progress: kbcli cluster describe-ops minio-ocqhov-volumeexpansion-bftrm -n ns-rkjff check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-volumeexpansion-bftrm ns-rkjff VolumeExpansion minio-ocqhov minio Creating -/- Sep 01,2025 11:38 UTC+0800 check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Updating Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:35 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:36 UTC+0800 minio-ocqhov-minio-2 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:36 UTC+0800 minio-ocqhov-minio-3 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:37 UTC+0800 check pod status done connect unsupported engine type: minio minio No resources found in minio-ocqhov namespace. check ops status `kbcli cluster list-ops minio-ocqhov --status all --namespace ns-rkjff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME minio-ocqhov-volumeexpansion-bftrm ns-rkjff VolumeExpansion minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:38 UTC+0800 check ops status done ops_status:minio-ocqhov-volumeexpansion-bftrm ns-rkjff VolumeExpansion minio-ocqhov minio Succeed 4/4 Sep 01,2025 11:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests minio-ocqhov-volumeexpansion-bftrm --namespace ns-rkjff ` opsrequest.apps.kubeblocks.io/minio-ocqhov-volumeexpansion-bftrm patched `kbcli cluster delete-ops --name minio-ocqhov-volumeexpansion-bftrm --force --auto-approve --namespace ns-rkjff ` OpsRequest minio-ocqhov-volumeexpansion-bftrm deleted `kubectl get secrets -l app.kubernetes.io/instance=minio-ocqhov` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.username***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.password***"` `kubectl get secrets minio-ocqhov-minio-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:CI89nk94g56V09eN;DB_PORT:9000;DB_DATABASE: check db_client batch data count `echo "mc config host add minioserver http://minio-ocqhov-minio.ns-rkjff.svc.cluster.local:9000 root CI89nk94g56V09eN;mc cat minioserver/executions-loop-bucket/executions_loop" | kubectl exec -it minio-ocqhov-minio-0 --namespace ns-rkjff -- bash` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update minio-ocqhov --termination-policy=WipeOut --namespace ns-rkjff ` cluster.apps.kubeblocks.io/minio-ocqhov updated (no change) check cluster status `kbcli cluster list minio-ocqhov --show-labels --namespace ns-rkjff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS minio-ocqhov ns-rkjff WipeOut Running Sep 01,2025 11:18 UTC+0800 app.kubernetes.io/instance=minio-ocqhov check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances minio-ocqhov --namespace ns-rkjff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME minio-ocqhov-minio-0 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:35 UTC+0800 minio-ocqhov-minio-1 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:36 UTC+0800 minio-ocqhov-minio-2 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:36 UTC+0800 minio-ocqhov-minio-3 ns-rkjff minio-ocqhov minio Running readwrite 0 200m / 200m 644245094400m / 644245094400m data:5Gi aks-cicdamdpool-18448605-vmss000001/10.224.0.5 Sep 01,2025 11:37 UTC+0800 check pod status done connect unsupported engine type: minio minio cluster list-logs `kbcli cluster list-logs minio-ocqhov --namespace ns-rkjff ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update minio-ocqhov --enable-all-logs=true --namespace ns-rkjff Error from server (NotFound): pods "minio-ocqhov-minio-0" not found cluster logs `kbcli cluster logs minio-ocqhov --tail 30 --namespace ns-rkjff ` Defaulted container "minio" out of: minio, lorry, init (init), init-lorry (init) 6: internal/logger/logger.go:268:logger.LogIf() 5: cmd/logging.go:156:cmd.storageLogIf() 4: cmd/erasure-server-pool.go:2604:cmd.(*erasureServerPools).Health() 3: cmd/server-main.go:924:cmd.serverMain.func11() 2: cmd/server-main.go:561:cmd.bootstrapTrace() 1: cmd/server-main.go:923:cmd.serverMain() Waiting for all MinIO sub-systems to be initialize... Configured max API requests per node based on available memory: 136 All MinIO sub-systems initialized successfully in 7.839964ms MinIO Object Storage Server Copyright: 2015-2025 MinIO, Inc. License: GNU AGPLv3 - https://www.gnu.org/licenses/agpl-3.0.html Version: RELEASE.2024-06-29T01-20-47Z (go1.22.4 linux/amd64) API: http://10.244.1.43:9000 http://127.0.0.1:9000 WebUI: http://10.244.1.43:9001 http://127.0.0.1:9001 Docs: https://min.io/docs/minio/linux/index.html Use `mc admin info` to look for latest server/drive info Status: 2 Online, 2 Offline. API: SYSTEM.peers Time: 03:37:16 UTC 09/01/2025 DeploymentID: bb065c59-6d0e-4cfb-8f58-ec05ded26ac4 Error: Drive: http://minio-ocqhov-minio-3.minio-ocqhov-minio-headless.ns-rkjff.svc.cluster.local:9000/data returned drive not found (*fmt.wrapError) endpoint="http://minio-ocqhov-minio-3.minio-ocqhov-minio-headless.ns-rkjff.svc.cluster.local:9000/data" 4: internal/logger/logger.go:258:logger.LogAlwaysIf() 3: cmd/logging.go:65:cmd.peersLogAlwaysIf() 2: cmd/prepare-storage.go:51:cmd.init.func22.1() 1: cmd/erasure-sets.go:234:cmd.(*erasureSets).connectDisks.func2() delete cluster minio-ocqhov `kbcli cluster delete minio-ocqhov --auto-approve --namespace ns-rkjff ` Cluster minio-ocqhov deleted pod_info:minio-ocqhov-minio-0 2/2 Running 0 7m6s minio-ocqhov-minio-1 2/2 Running 0 6m9s minio-ocqhov-minio-2 2/2 Running 0 5m33s minio-ocqhov-minio-3 2/2 Running 0 4m59s No resources found in ns-rkjff namespace. delete cluster pod done No resources found in ns-rkjff namespace. check cluster resource non-exist OK: pvc No resources found in ns-rkjff namespace. delete cluster done No resources found in ns-rkjff namespace. No resources found in ns-rkjff namespace. No resources found in ns-rkjff namespace. Minio Test Suite All Done! --------------------------------------Minio (Topology = Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=minio;]|[Description=Create a cluster with the specified cluster definition minio] [PASSED]|[VerticalScaling]|[ComponentName=minio]|[Description=VerticalScaling the cluster specify component minio] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=minio]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Bench]|[ComponentName=minio]|[Description=Bench the cluster service with minio component] [PASSED]|[HorizontalScaling Out]|[ComponentName=minio]|[Description=HorizontalScaling Out the cluster specify component minio] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=minio]|[Description=VolumeExpansion the cluster specify component minio] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]