bash test/kbcli/test_kbcli_0.9.sh --type 57 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-fkhrb ` `kubectl create namespace ns-fkhrb` namespace/ns-fkhrb created create namespace ns-fkhrb done check registry secret No resources found in ns-fkhrb namespace. create docker registry server https://index.docker.io/v1/ secret/kbcli-test-registry-key created create docker registry secret kbcli-test-registry-key Success check ns:ns-fkhrb sa:default imagePullSecrets patch ns:ns-fkhrb sa:default imagePullSecrets serviceaccount/default patched patch ns:ns-fkhrb sa:default imagePullSecrets Success download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 56 32.1M 56 18.2M 0 0 64.5M 0 --:--:-- --:--:-- --:--:-- 64.5M 100 32.1M 100 32.1M 0 0 98.1M 0 --:--:-- --:--:-- --:--:-- 310M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:dmdb set component version set component version:damengdb set service versions:8.1.0,8.1.3-100-20240124,8.1.3-26-20240821,8.1.4-6-20241231 set service versions sorted:8.1.0,8.1.3-100-20240124,8.1.3-26-20240821,8.1.4-6-20241231 no cluster version found set damengdb component definition set damengdb component definition dmdb-8.1 REPORT_COUNT 0:0 set replicas first:3,8.1.0|3,8.1.3-100-20240124|3,8.1.3-26-20240821 set replicas third:3,8.1.3-100-20240124 set replicas fourth:3,8.1.0 set minimum cmpv service version set minimum cmpv service version replicas:3,8.1.3-26-20240821 REPORT_COUNT:1 CLUSTER_TOPOLOGY:realtime-replication topology realtime-replication found in cluster definition damengdb LIMIT_CPU:1 LIMIT_MEMORY:2 storage size: 20 No resources found in ns-fkhrb namespace. termination_policy:DoNotTerminate create 3 replica DoNotTerminate damengdb cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition1: dmdb-8.1 by component version:damengdb apiVersion: v1 kind: Secret metadata: name: damengdb-tkwqqh-license namespace: ns-fkhrb type: Opaque --- apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: damengdb-tkwqqh namespace: ns-fkhrb spec: clusterDefinitionRef: damengdb topology: realtime-replication terminationPolicy: DoNotTerminate componentSpecs: - name: dmdb serviceVersion: 8.1.3-26-20240821 podUpdatePolicy: Recreate replicas: 3 env: - name: "OGUID" value: "1" - name: "UNICODE_FLAG" value: "1" - name: "LENGTH_IN_CHAR" value: "1" - name: "CASE_SENSITIVE" value: "0" - name: "BLANK_PAD_MODE" value: "0" - name: "ARCH_WAIT_APPLY" value: "0" - name: "HOST_NETWORK" value: "disabled" - name: "TIME_ZONE" value: "+08:00" - name: "ENABLE_AUDIT" value: "true" - name: "INSTANCE_NAME_PREFIX" value: "" - name: "PAGE_SIZE" value: "8" - name: "EXTENT_SIZE" value: "16" - name: "LOG_SIZE" value: "2048" - name: "SORT_MODE" value: "0" - name: "PK_CLUSTER_MODE" value: "0" - name: "ARCH_SPACE_LIMIT" value: "51200" - name: "IS_REMOTE_BACKUP" value: "false" - name: "REMOTE_TIMER_INTERVAL" value: "1" volumes: - name: license secret: secretName: damengdb-tkwqqh-license resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: arch spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: dmmonitor serviceVersion: 8.1.3-26-20240821 podUpdatePolicy: Recreate replicas: 1 env: - name: "OGUID" value: "1" resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi `kubectl apply -f test_create_damengdb-tkwqqh.yaml` secret/damengdb-tkwqqh-license created cluster.apps.kubeblocks.io/damengdb-tkwqqh created apply test_create_damengdb-tkwqqh.yaml Success `rm -rf test_create_damengdb-tkwqqh.yaml` `kubectl get serviceaccount --no-headers --namespace ns-fkhrb -l app.kubernetes.io/instance=damengdb-tkwqqh` patch ns:ns-fkhrb sa:kb-damengdb-tkwqqh imagePullSecrets serviceaccount/kb-damengdb-tkwqqh patched patch ns:ns-fkhrb sa:kb-damengdb-tkwqqh imagePullSecrets Success check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Creating Sep 01,2025 11:24 UTC+0800 clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba check pod damengdb-tkwqqh-dmdb-0 container_name dmdb exist password ZUAH8w9u6i check pod damengdb-tkwqqh-dmdb-0 container_name exporter exist password ZUAH8w9u6i check pod damengdb-tkwqqh-dmdb-0 container_name lorry exist password ZUAH8w9u6i check pod damengdb-tkwqqh-dmdb-0 container_name config-manager exist password ZUAH8w9u6i No container logs contain secret password. describe cluster `kbcli cluster describe damengdb-tkwqqh --namespace ns-fkhrb ` Name: damengdb-tkwqqh Created Time: Sep 01,2025 11:24 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-fkhrb damengdb Running DoNotTerminate Endpoints: COMPONENT MODE INTERNAL EXTERNAL dmdb ReadWrite damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME dmdb damengdb-tkwqqh-dmdb-0 primary Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 dmdb damengdb-tkwqqh-dmdb-1 standby Running 0 aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 dmdb damengdb-tkwqqh-dmdb-2 standby Running 0 aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 dmmonitor damengdb-tkwqqh-dmmonitor-0 Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS dmdb false 1 / 1 2Gi / 2Gi data:20Gi default arch:20Gi default log:20Gi default dmmonitor false 1 / 1 2Gi / 2Gi data:20Gi default Images: COMPONENT TYPE IMAGE dmdb docker.io/apecloud/dm:8.1.3-26 dmmonitor docker.io/apecloud/dm:8.1.3-26 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-fkhrb damengdb-tkwqqh `kbcli cluster label damengdb-tkwqqh app.kubernetes.io/instance- --namespace ns-fkhrb ` label "app.kubernetes.io/instance" not found. `kbcli cluster label damengdb-tkwqqh app.kubernetes.io/instance=damengdb-tkwqqh --namespace ns-fkhrb ` `kbcli cluster label damengdb-tkwqqh --list --namespace ns-fkhrb ` NAME NAMESPACE LABELS damengdb-tkwqqh ns-fkhrb app.kubernetes.io/instance=damengdb-tkwqqh clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=damengdb-tkwqqh Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=damengdb-tkwqqh --namespace ns-fkhrb ` `kbcli cluster label damengdb-tkwqqh --list --namespace ns-fkhrb ` NAME NAMESPACE LABELS damengdb-tkwqqh ns-fkhrb app.kubernetes.io/instance=damengdb-tkwqqh case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label damengdb-tkwqqh case.name=kbcli.test2 --overwrite --namespace ns-fkhrb ` `kbcli cluster label damengdb-tkwqqh --list --namespace ns-fkhrb ` NAME NAMESPACE LABELS damengdb-tkwqqh ns-fkhrb app.kubernetes.io/instance=damengdb-tkwqqh case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label damengdb-tkwqqh case.name- --namespace ns-fkhrb ` `kbcli cluster label damengdb-tkwqqh --list --namespace ns-fkhrb ` NAME NAMESPACE LABELS damengdb-tkwqqh ns-fkhrb app.kubernetes.io/instance=damengdb-tkwqqh clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect `echo 'cat /opt/dmdbms/data/DAMENG/dm.ini | grep MAX_SESSIONS' | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash ` Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file MAX_SESSIONS = 1500 #Maximum number of concurrent sessions connect cluster Success set max_connections to 1500 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-damengdb-tkwqqh" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-damengdb-tkwqqh --namespace ns-fkhrb ` Error from server (NotFound): pods "test-db-client-executionloop-damengdb-tkwqqh" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-damengdb-tkwqqh" not found `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-damengdb-tkwqqh namespace: ns-fkhrb spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local" - "--user" - "sysdba" - "--password" - "ZUAH8w9u6i" - "--port" - "5236" - "--dbtype" - "damengdb" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-damengdb-tkwqqh.yaml` pod/test-db-client-executionloop-damengdb-tkwqqh created apply test-db-client-executionloop-damengdb-tkwqqh.yaml Success `rm -rf test-db-client-executionloop-damengdb-tkwqqh.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 21s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 26s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 31s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 36s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 42s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 52s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 1/1 Running 0 63s check pod test-db-client-executionloop-damengdb-tkwqqh status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-tkwqqh 0/1 Completed 0 68s check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done --host damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local --user sysdba --password ZUAH8w9u6i --port 5236 --dbtype damengdb --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Execution loop start: tableCount: 0 create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); Execution loop start:INSERT INTO executions_loop_table (value) VALUES ('executions_loop_test_1'); [ 1s ] executions total: 176 successful: 176 failed: 0 disconnect: 0 [ 2s ] executions total: 364 successful: 364 failed: 0 disconnect: 0 [ 3s ] executions total: 558 successful: 558 failed: 0 disconnect: 0 [ 4s ] executions total: 756 successful: 756 failed: 0 disconnect: 0 [ 5s ] executions total: 950 successful: 950 failed: 0 disconnect: 0 [ 6s ] executions total: 1139 successful: 1139 failed: 0 disconnect: 0 [ 7s ] executions total: 1328 successful: 1328 failed: 0 disconnect: 0 [ 8s ] executions total: 1509 successful: 1509 failed: 0 disconnect: 0 [ 9s ] executions total: 1701 successful: 1701 failed: 0 disconnect: 0 [ 10s ] executions total: 1891 successful: 1891 failed: 0 disconnect: 0 [ 11s ] executions total: 2056 successful: 2056 failed: 0 disconnect: 0 [ 12s ] executions total: 2241 successful: 2241 failed: 0 disconnect: 0 [ 13s ] executions total: 2436 successful: 2436 failed: 0 disconnect: 0 [ 14s ] executions total: 2633 successful: 2633 failed: 0 disconnect: 0 [ 15s ] executions total: 2831 successful: 2831 failed: 0 disconnect: 0 [ 16s ] executions total: 3027 successful: 3027 failed: 0 disconnect: 0 [ 17s ] executions total: 3222 successful: 3222 failed: 0 disconnect: 0 [ 18s ] executions total: 3418 successful: 3418 failed: 0 disconnect: 0 [ 19s ] executions total: 3613 successful: 3613 failed: 0 disconnect: 0 [ 20s ] executions total: 3808 successful: 3808 failed: 0 disconnect: 0 [ 21s ] executions total: 4004 successful: 4004 failed: 0 disconnect: 0 [ 22s ] executions total: 4198 successful: 4198 failed: 0 disconnect: 0 [ 23s ] executions total: 4380 successful: 4380 failed: 0 disconnect: 0 [ 24s ] executions total: 4564 successful: 4564 failed: 0 disconnect: 0 [ 25s ] executions total: 4737 successful: 4737 failed: 0 disconnect: 0 [ 26s ] executions total: 4930 successful: 4930 failed: 0 disconnect: 0 [ 27s ] executions total: 5122 successful: 5122 failed: 0 disconnect: 0 [ 28s ] executions total: 5308 successful: 5308 failed: 0 disconnect: 0 [ 29s ] executions total: 5490 successful: 5490 failed: 0 disconnect: 0 [ 30s ] executions total: 5679 successful: 5679 failed: 0 disconnect: 0 [ 31s ] executions total: 5867 successful: 5867 failed: 0 disconnect: 0 [ 32s ] executions total: 6062 successful: 6062 failed: 0 disconnect: 0 [ 33s ] executions total: 6262 successful: 6262 failed: 0 disconnect: 0 [ 34s ] executions total: 6458 successful: 6458 failed: 0 disconnect: 0 [ 35s ] executions total: 6649 successful: 6649 failed: 0 disconnect: 0 [ 36s ] executions total: 6840 successful: 6840 failed: 0 disconnect: 0 [ 37s ] executions total: 7012 successful: 7012 failed: 0 disconnect: 0 [ 38s ] executions total: 7190 successful: 7190 failed: 0 disconnect: 0 [ 39s ] executions total: 7372 successful: 7372 failed: 0 disconnect: 0 [ 40s ] executions total: 7546 successful: 7546 failed: 0 disconnect: 0 [ 41s ] executions total: 7736 successful: 7736 failed: 0 disconnect: 0 [ 42s ] executions total: 7903 successful: 7903 failed: 0 disconnect: 0 [ 43s ] executions total: 8101 successful: 8101 failed: 0 disconnect: 0 [ 44s ] executions total: 8297 successful: 8297 failed: 0 disconnect: 0 [ 45s ] executions total: 8491 successful: 8491 failed: 0 disconnect: 0 [ 46s ] executions total: 8672 successful: 8672 failed: 0 disconnect: 0 [ 47s ] executions total: 8862 successful: 8862 failed: 0 disconnect: 0 [ 48s ] executions total: 9061 successful: 9061 failed: 0 disconnect: 0 [ 49s ] executions total: 9259 successful: 9259 failed: 0 disconnect: 0 [ 50s ] executions total: 9456 successful: 9456 failed: 0 disconnect: 0 [ 51s ] executions total: 9648 successful: 9648 failed: 0 disconnect: 0 [ 52s ] executions total: 9837 successful: 9837 failed: 0 disconnect: 0 [ 53s ] executions total: 10025 successful: 10025 failed: 0 disconnect: 0 [ 54s ] executions total: 10214 successful: 10214 failed: 0 disconnect: 0 [ 55s ] executions total: 10404 successful: 10404 failed: 0 disconnect: 0 [ 56s ] executions total: 10595 successful: 10595 failed: 0 disconnect: 0 [ 57s ] executions total: 10788 successful: 10788 failed: 0 disconnect: 0 [ 58s ] executions total: 10986 successful: 10986 failed: 0 disconnect: 0 [ 59s ] executions total: 11178 successful: 11178 failed: 0 disconnect: 0 [ 60s ] executions total: 11345 successful: 11345 failed: 0 disconnect: 0 Test Result: Total Executions: 11345 Successful Executions: 11345 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: damengdb Host: damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local Port: 5236 Database: Table: User: sysdba Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 11345 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-damengdb-tkwqqh --namespace ns-fkhrb ` pod/test-db-client-executionloop-damengdb-tkwqqh patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-damengdb-tkwqqh" force deleted test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh created apply test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test switchover cluster promote apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-tkwqqh-switchover- namespace: ns-fkhrb spec: clusterName: damengdb-tkwqqh custom: components: - componentName: dmmonitor parameters: - name: candidate value: damengdb-tkwqqh-dmdb-0 opsDefinitionName: switchover-for-dm type: Custom check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-tkwqqh.yaml` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-switchover-wzdb4 created create test_ops_cluster_damengdb-tkwqqh.yaml Success `rm -rf test_ops_cluster_damengdb-tkwqqh.yaml` check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-switchover-wzdb4 ns-fkhrb Custom damengdb-tkwqqh Running -/- Sep 01,2025 11:35 UTC+0800 ops_status:damengdb-tkwqqh-switchover-wzdb4 ns-fkhrb Custom damengdb-tkwqqh Failed -/- Sep 01,2025 11:35 UTC+0800 ops Switchover|Custom Succeed or Failed Soon check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-switchover-wzdb4 ns-fkhrb Custom damengdb-tkwqqh Failed -/- Sep 01,2025 11:35 UTC+0800 check ops status done check opsrequest progress test/kbcli/test_kbcli_0.9/kubeblocks/resource_check.sh: line 1308: [[: -: syntax error: operand expected (error token is "-") --------------------------------------get cluster damengdb-tkwqqh yaml-------------------------------------- `kubectl get cluster damengdb-tkwqqh -o yaml --namespace ns-fkhrb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-09-01T03:34:24.487406168Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"damengdb-tkwqqh","namespace":"ns-fkhrb"***,"spec":***"clusterDefinitionRef":"damengdb","componentSpecs":[***"env":[***"name":"OGUID","value":"1"***,***"name":"UNICODE_FLAG","value":"1"***,***"name":"LENGTH_IN_CHAR","value":"1"***,***"name":"CASE_SENSITIVE","value":"0"***,***"name":"BLANK_PAD_MODE","value":"0"***,***"name":"ARCH_WAIT_APPLY","value":"0"***,***"name":"HOST_NETWORK","value":"disabled"***,***"name":"TIME_ZONE","value":"+08:00"***,***"name":"ENABLE_AUDIT","value":"true"***,***"name":"INSTANCE_NAME_PREFIX","value":""***,***"name":"PAGE_SIZE","value":"8"***,***"name":"EXTENT_SIZE","value":"16"***,***"name":"LOG_SIZE","value":"2048"***,***"name":"SORT_MODE","value":"0"***,***"name":"PK_CLUSTER_MODE","value":"0"***,***"name":"ARCH_SPACE_LIMIT","value":"51200"***,***"name":"IS_REMOTE_BACKUP","value":"false"***,***"name":"REMOTE_TIMER_INTERVAL","value":"1"***],"name":"dmdb","podUpdatePolicy":"Recreate","replicas":3,"resources":***"limits":***"cpu":"1000m","memory":"2Gi"***,"requests":***"cpu":"1000m","memory":"2Gi"***,"serviceVersion":"8.1.3-26-20240821","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***,***"name":"arch","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***,***"name":"log","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***],"volumes":[***"name":"license","secret":***"secretName":"damengdb-tkwqqh-license"***]***,***"env":[***"name":"OGUID","value":"1"***],"name":"dmmonitor","podUpdatePolicy":"Recreate","replicas":1,"resources":***"limits":***"cpu":"1000m","memory":"2Gi"***,"requests":***"cpu":"1000m","memory":"2Gi"***,"serviceVersion":"8.1.3-26-20240821","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***]***],"terminationPolicy":"DoNotTerminate","topology":"realtime-replication"*** creationTimestamp: "2025-09-01T03:24:02Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 2 labels: app.kubernetes.io/instance: damengdb-tkwqqh clusterdefinition.kubeblocks.io/name: damengdb clusterversion.kubeblocks.io/name: "" name: damengdb-tkwqqh namespace: ns-fkhrb resourceVersion: "28132" uid: c1f73216-28ad-4a1c-b972-dee7b6f5fe86 spec: clusterDefinitionRef: damengdb componentSpecs: - componentDef: dmdb-8.1 env: - name: OGUID value: "1" - name: UNICODE_FLAG value: "1" - name: LENGTH_IN_CHAR value: "1" - name: CASE_SENSITIVE value: "0" - name: BLANK_PAD_MODE value: "0" - name: ARCH_WAIT_APPLY value: "0" - name: HOST_NETWORK value: disabled - name: TIME_ZONE value: "+08:00" - name: ENABLE_AUDIT value: "true" - name: INSTANCE_NAME_PREFIX - name: PAGE_SIZE value: "8" - name: EXTENT_SIZE value: "16" - name: LOG_SIZE value: "2048" - name: SORT_MODE value: "0" - name: PK_CLUSTER_MODE value: "0" - name: ARCH_SPACE_LIMIT value: "51200" - name: IS_REMOTE_BACKUP value: "false" - name: REMOTE_TIMER_INTERVAL value: "1" name: dmdb podUpdatePolicy: Recreate replicas: 3 resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: arch spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumes: - name: license secret: secretName: damengdb-tkwqqh-license - componentDef: dmmonitor-8.1 env: - name: OGUID value: "1" name: dmmonitor podUpdatePolicy: Recreate replicas: 1 resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi terminationPolicy: DoNotTerminate topology: realtime-replication status: clusterDefGeneration: 2 components: dmdb: phase: Running podsReady: true podsReadyTime: "2025-09-01T03:35:40Z" dmmonitor: phase: Running podsReady: true podsReadyTime: "2025-09-01T03:26:28Z" conditions: - lastTransitionTime: "2025-09-01T03:24:02Z" message: 'The operator has started the provisioning of Cluster: damengdb-tkwqqh' observedGeneration: 2 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:24:02Z" message: Successfully applied for resources observedGeneration: 2 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T03:35:40Z" message: all pods of components are ready, waiting for the probe detection successful reason: AllReplicasReady status: "True" type: ReplicasReady - lastTransitionTime: "2025-09-01T03:35:40Z" message: 'Cluster: damengdb-tkwqqh is ready, current phase is Running' reason: ClusterReady status: "True" type: Ready observedGeneration: 2 phase: Running ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster damengdb-tkwqqh-------------------------------------- `kubectl describe cluster damengdb-tkwqqh --namespace ns-fkhrb ` Name: damengdb-tkwqqh Namespace: ns-fkhrb Labels: app.kubernetes.io/instance=damengdb-tkwqqh clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-09-01T03:34:24.487406168Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:24:02Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 2 Resource Version: 28132 UID: c1f73216-28ad-4a1c-b972-dee7b6f5fe86 Spec: Cluster Definition Ref: damengdb Component Specs: Component Def: dmdb-8.1 Env: Name: OGUID Value: 1 Name: UNICODE_FLAG Value: 1 Name: LENGTH_IN_CHAR Value: 1 Name: CASE_SENSITIVE Value: 0 Name: BLANK_PAD_MODE Value: 0 Name: ARCH_WAIT_APPLY Value: 0 Name: HOST_NETWORK Value: disabled Name: TIME_ZONE Value: +08:00 Name: ENABLE_AUDIT Value: true Name: INSTANCE_NAME_PREFIX Name: PAGE_SIZE Value: 8 Name: EXTENT_SIZE Value: 16 Name: LOG_SIZE Value: 2048 Name: SORT_MODE Value: 0 Name: PK_CLUSTER_MODE Value: 0 Name: ARCH_SPACE_LIMIT Value: 51200 Name: IS_REMOTE_BACKUP Value: false Name: REMOTE_TIMER_INTERVAL Value: 1 Name: dmdb Pod Update Policy: Recreate Replicas: 3 Resources: Limits: Cpu: 1 Memory: 2Gi Requests: Cpu: 1 Memory: 2Gi Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: arch Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Volumes: Name: license Secret: Secret Name: damengdb-tkwqqh-license Component Def: dmmonitor-8.1 Env: Name: OGUID Value: 1 Name: dmmonitor Pod Update Policy: Recreate Replicas: 1 Resources: Limits: Cpu: 1 Memory: 2Gi Requests: Cpu: 1 Memory: 2Gi Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Termination Policy: DoNotTerminate Topology: realtime-replication Status: Cluster Def Generation: 2 Components: Dmdb: Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:35:40Z Dmmonitor: Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:26:28Z Conditions: Last Transition Time: 2025-09-01T03:24:02Z Message: The operator has started the provisioning of Cluster: damengdb-tkwqqh Observed Generation: 2 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:24:02Z Message: Successfully applied for resources Observed Generation: 2 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T03:35:40Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2025-09-01T03:35:40Z Message: Cluster: damengdb-tkwqqh is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 2 Phase: Running Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 12m (x2 over 12m) cluster-controller The operator has started the provisioning of Cluster: damengdb-tkwqqh Normal ApplyResourcesSucceed 12m (x2 over 12m) cluster-controller Successfully applied for resources Normal ComponentPhaseTransition 12m (x2 over 12m) cluster-controller component is Creating Warning ReplicasNotReady 2m43s (x2 over 9m48s) cluster-controller pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Warning ComponentsNotReady 2m43s (x2 over 9m48s) cluster-controller pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 2m43s cluster-controller component is Updating Warning Unhealthy 115s (x4 over 2m14s) event-controller Pod damengdb-tkwqqh-dmdb-0: Readiness probe failed: Get "http://10.244.5.239:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 112s event-controller Pod damengdb-tkwqqh-dmdb-0: Readiness probe failed: Get "http://10.244.5.239:3501/v1.0/checkrole": dial tcp 10.244.5.239:3501: connect: no route to host Normal ComponentPhaseTransition 36s (x3 over 9m48s) cluster-controller component is Running Normal AllReplicasReady 36s (x2 over 4m36s) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 36s (x2 over 4m36s) cluster-controller Cluster: damengdb-tkwqqh is ready, current phase is Running Normal Running 36s (x2 over 4m36s) cluster-controller Cluster: damengdb-tkwqqh is ready, current phase is Running ------------------------------------------------------------------------------------------------------------------ ops_status:damengdb-tkwqqh-switchover-wzdb4 ns-fkhrb Custom damengdb-tkwqqh Failed -/- Sep 01,2025 11:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-switchover-wzdb4 --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-switchover-wzdb4 patched `kbcli cluster delete-ops --name damengdb-tkwqqh-switchover-wzdb4 --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-switchover-wzdb4 deleted switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... switchover pod:damengdb-tkwqqh-dmdb-1 checking switchover... check switchover timeout test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-damengdb-tkwqqh created apply test-chaos-mesh-networkdelay-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networkdelay-damengdb-tkwqqh.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-tkwqqh" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-damengdb-tkwqqh patched check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover drainnode check cluster status before cluster-failover-drainnode check cluster status done cluster_status:Running check node drain check node drain success kubectl get pod damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-38368475-vmss000006 success check if multiple pods are on the same node kubectl get pod damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-38368475-vmss000000 success kubectl get pod damengdb-tkwqqh-dmdb-2 --namespace ns-fkhrb -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-38368475-vmss000006 success Multiple pods on the same node test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-damengdb-tkwqqh created apply test-chaos-mesh-networkduplicate-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networkduplicate-damengdb-tkwqqh.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (InternalError): Internal error occurred: failed calling webhook "mnetworkchaos.kb.io": failed to call webhook: Post "https://chaos-mesh-controller-manager.chaos-mesh.svc:443/mutate-chaos-mesh-org-v1alpha1-networkchaos?timeout=5s": no endpoints available for service "chaos-mesh-controller-manager" check failover pod name networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-tkwqqh" force deleted failover pod name:damengdb-tkwqqh-dmdb-1 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-2 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-damengdb-tkwqqh created apply test-chaos-mesh-networkcorruptover-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-damengdb-tkwqqh.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-damengdb-tkwqqh created apply test-chaos-mesh-networkcorruptover-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-damengdb-tkwqqh.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:27 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-2;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-2 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-tkwqqh" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-damengdb-tkwqqh patched check failover pod name:damengdb-tkwqqh-dmdb-2 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-2 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale damengdb-tkwqqh --auto-approve --force=true --components dmdb --cpu 1100m --memory 2.1Gi --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-verticalscaling-x9tlt created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-verticalscaling-x9tlt -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Creating -/- Sep 01,2025 11:51 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster damengdb-tkwqqh yaml-------------------------------------- `kubectl get cluster damengdb-tkwqqh -o yaml --namespace ns-fkhrb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"damengdb-tkwqqh-verticalscaling-x9tlt","type":"VerticalScaling"***]' kubeblocks.io/reconcile: "2025-09-01T03:56:05.593049042Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"damengdb-tkwqqh","namespace":"ns-fkhrb"***,"spec":***"clusterDefinitionRef":"damengdb","componentSpecs":[***"env":[***"name":"OGUID","value":"1"***,***"name":"UNICODE_FLAG","value":"1"***,***"name":"LENGTH_IN_CHAR","value":"1"***,***"name":"CASE_SENSITIVE","value":"0"***,***"name":"BLANK_PAD_MODE","value":"0"***,***"name":"ARCH_WAIT_APPLY","value":"0"***,***"name":"HOST_NETWORK","value":"disabled"***,***"name":"TIME_ZONE","value":"+08:00"***,***"name":"ENABLE_AUDIT","value":"true"***,***"name":"INSTANCE_NAME_PREFIX","value":""***,***"name":"PAGE_SIZE","value":"8"***,***"name":"EXTENT_SIZE","value":"16"***,***"name":"LOG_SIZE","value":"2048"***,***"name":"SORT_MODE","value":"0"***,***"name":"PK_CLUSTER_MODE","value":"0"***,***"name":"ARCH_SPACE_LIMIT","value":"51200"***,***"name":"IS_REMOTE_BACKUP","value":"false"***,***"name":"REMOTE_TIMER_INTERVAL","value":"1"***],"name":"dmdb","podUpdatePolicy":"Recreate","replicas":3,"resources":***"limits":***"cpu":"1000m","memory":"2Gi"***,"requests":***"cpu":"1000m","memory":"2Gi"***,"serviceVersion":"8.1.3-26-20240821","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***,***"name":"arch","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***,***"name":"log","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***],"volumes":[***"name":"license","secret":***"secretName":"damengdb-tkwqqh-license"***]***,***"env":[***"name":"OGUID","value":"1"***],"name":"dmmonitor","podUpdatePolicy":"Recreate","replicas":1,"resources":***"limits":***"cpu":"1000m","memory":"2Gi"***,"requests":***"cpu":"1000m","memory":"2Gi"***,"serviceVersion":"8.1.3-26-20240821","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,"storageClassName":null***]***],"terminationPolicy":"DoNotTerminate","topology":"realtime-replication"*** creationTimestamp: "2025-09-01T03:24:02Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 4 labels: app.kubernetes.io/instance: damengdb-tkwqqh clusterdefinition.kubeblocks.io/name: damengdb clusterversion.kubeblocks.io/name: "" name: damengdb-tkwqqh namespace: ns-fkhrb resourceVersion: "49182" uid: c1f73216-28ad-4a1c-b972-dee7b6f5fe86 spec: clusterDefinitionRef: damengdb componentSpecs: - componentDef: dmdb-8.1 env: - name: OGUID value: "1" - name: UNICODE_FLAG value: "1" - name: LENGTH_IN_CHAR value: "1" - name: CASE_SENSITIVE value: "0" - name: BLANK_PAD_MODE value: "0" - name: ARCH_WAIT_APPLY value: "0" - name: HOST_NETWORK value: disabled - name: TIME_ZONE value: "+08:00" - name: ENABLE_AUDIT value: "true" - name: INSTANCE_NAME_PREFIX - name: PAGE_SIZE value: "8" - name: EXTENT_SIZE value: "16" - name: LOG_SIZE value: "2048" - name: SORT_MODE value: "0" - name: PK_CLUSTER_MODE value: "0" - name: ARCH_SPACE_LIMIT value: "51200" - name: IS_REMOTE_BACKUP value: "false" - name: REMOTE_TIMER_INTERVAL value: "1" name: dmdb podUpdatePolicy: Recreate replicas: 3 resources: limits: cpu: 1100m memory: 2254857830400m requests: cpu: 1100m memory: 2254857830400m serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: arch spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumes: - name: license secret: secretName: damengdb-tkwqqh-license - componentDef: dmmonitor-8.1 env: - name: OGUID value: "1" name: dmmonitor podUpdatePolicy: Recreate replicas: 1 resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: DoNotTerminate topology: realtime-replication status: clusterDefGeneration: 2 components: dmdb: phase: Updating podsReady: false podsReadyTime: "2025-09-01T03:51:14Z" dmmonitor: phase: Running podsReady: true podsReadyTime: "2025-09-01T03:46:45Z" conditions: - lastTransitionTime: "2025-09-01T03:24:02Z" message: 'The operator has started the provisioning of Cluster: damengdb-tkwqqh' observedGeneration: 4 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:24:02Z" message: Successfully applied for resources observedGeneration: 4 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T03:51:32Z" message: 'pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T03:51:32Z" message: 'pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 4 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster damengdb-tkwqqh-------------------------------------- `kubectl describe cluster damengdb-tkwqqh --namespace ns-fkhrb ` Name: damengdb-tkwqqh Namespace: ns-fkhrb Labels: app.kubernetes.io/instance=damengdb-tkwqqh clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"damengdb-tkwqqh-verticalscaling-x9tlt","type":"VerticalScaling"***] kubeblocks.io/reconcile: 2025-09-01T03:56:05.593049042Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:24:02Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 4 Resource Version: 49182 UID: c1f73216-28ad-4a1c-b972-dee7b6f5fe86 Spec: Cluster Definition Ref: damengdb Component Specs: Component Def: dmdb-8.1 Env: Name: OGUID Value: 1 Name: UNICODE_FLAG Value: 1 Name: LENGTH_IN_CHAR Value: 1 Name: CASE_SENSITIVE Value: 0 Name: BLANK_PAD_MODE Value: 0 Name: ARCH_WAIT_APPLY Value: 0 Name: HOST_NETWORK Value: disabled Name: TIME_ZONE Value: +08:00 Name: ENABLE_AUDIT Value: true Name: INSTANCE_NAME_PREFIX Name: PAGE_SIZE Value: 8 Name: EXTENT_SIZE Value: 16 Name: LOG_SIZE Value: 2048 Name: SORT_MODE Value: 0 Name: PK_CLUSTER_MODE Value: 0 Name: ARCH_SPACE_LIMIT Value: 51200 Name: IS_REMOTE_BACKUP Value: false Name: REMOTE_TIMER_INTERVAL Value: 1 Name: dmdb Pod Update Policy: Recreate Replicas: 3 Resources: Limits: Cpu: 1100m Memory: 2254857830400m Requests: Cpu: 1100m Memory: 2254857830400m Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: arch Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Volumes: Name: license Secret: Secret Name: damengdb-tkwqqh-license Component Def: dmmonitor-8.1 Env: Name: OGUID Value: 1 Name: dmmonitor Pod Update Policy: Recreate Replicas: 1 Resources: Limits: Cpu: 1 Memory: 2Gi Requests: Cpu: 1 Memory: 2Gi Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: DoNotTerminate Topology: realtime-replication Status: Cluster Def Generation: 2 Components: Dmdb: Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T03:51:14Z Dmmonitor: Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:46:45Z Conditions: Last Transition Time: 2025-09-01T03:24:02Z Message: The operator has started the provisioning of Cluster: damengdb-tkwqqh Observed Generation: 4 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:24:02Z Message: Successfully applied for resources Observed Generation: 4 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T03:51:32Z Message: pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T03:51:32Z Message: pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 4 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ApplyResourcesSucceed 35m (x2 over 35m) cluster-controller Successfully applied for resources Normal PreCheckSucceed 35m (x2 over 35m) cluster-controller The operator has started the provisioning of Cluster: damengdb-tkwqqh Normal ComponentPhaseTransition 35m (x2 over 35m) cluster-controller component is Creating Warning Unhealthy 24m (x4 over 25m) event-controller Pod damengdb-tkwqqh-dmdb-0: Readiness probe failed: Get "http://10.244.5.239:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 24m event-controller Pod damengdb-tkwqqh-dmdb-0: Readiness probe failed: Get "http://10.244.5.239:3501/v1.0/checkrole": dial tcp 10.244.5.239:3501: connect: no route to host Normal ComponentPhaseTransition 19m (x2 over 25m) cluster-controller component is Updating Warning ReplicasNotReady 19m (x3 over 32m) cluster-controller pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Warning ComponentsNotReady 19m (x3 over 32m) cluster-controller pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Warning Unhealthy 18m (x21 over 19m) event-controller Pod damengdb-tkwqqh-dmdb-1: Readiness probe failed: Get "http://10.244.1.75:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal ComponentPhaseTransition 17m (x4 over 32m) cluster-controller component is Running Normal Running 17m (x3 over 27m) cluster-controller Cluster: damengdb-tkwqqh is ready, current phase is Running Normal ClusterReady 17m (x3 over 27m) cluster-controller Cluster: damengdb-tkwqqh is ready, current phase is Running Normal AllReplicasReady 17m (x3 over 27m) cluster-controller all pods of components are ready, waiting for the probe detection successful Warning Unhealthy 14m event-controller Pod damengdb-tkwqqh-dmdb-0: Readiness probe failed: Get "http://10.244.5.239:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning Unhealthy 14m event-controller Pod damengdb-tkwqqh-dmdb-0: Readiness probe failed: Get "http://10.244.5.239:3501/v1.0/checkrole": dial tcp 10.244.5.239:3501: connect: no route to host Warning ComponentsNotReady 14m cluster-controller pods are unavailable in Components: [dmdb dmmonitor], refer to related component message in Cluster.status.components Warning ReplicasNotReady 14m cluster-controller pods are not ready in Components: [dmdb dmmonitor], refer to related component message in Cluster.status.components Warning Unhealthy 8m55s (x3 over 14m) event-controller Pod damengdb-tkwqqh-dmdb-1: Readiness probe failed: Get "http://10.244.1.75:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Warning ReplicasNotReady 7m53s (x3 over 12m) cluster-controller pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 7m53s (x4 over 14m) cluster-controller component is Updating Warning ComponentsNotReady 7m53s (x3 over 12m) cluster-controller pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 7m51s (x4 over 12m) cluster-controller component is Running Normal AllReplicasReady 7m50s (x3 over 10m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 7m50s (x3 over 10m) cluster-controller Cluster: damengdb-tkwqqh is ready, current phase is Running Normal Running 7m50s (x3 over 10m) cluster-controller Cluster: damengdb-tkwqqh is ready, current phase is Running Normal PreCheckSucceed 7m35s cluster-controller The operator has started the provisioning of Cluster: damengdb-tkwqqh Warning FailedScheduling 3m25s (x2 over 13m) event-controller Pod damengdb-tkwqqh-dmmonitor-0: 0/3 nodes are available: 1 Insufficient memory, 1 node(s) were unschedulable, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod. Warning FailedScheduling 3m25s (x2 over 13m) event-controller Pod damengdb-tkwqqh-dmdb-0: 0/3 nodes are available: 1 Insufficient memory, 1 node(s) were unschedulable, 2 Insufficient cpu. preemption: 0/3 nodes are available: 1 Preemption is not helpful for scheduling, 2 No preemption victims found for incoming pod. Warning FailedAttachVolume 3m event-controller Pod damengdb-tkwqqh-dmdb-1: AttachVolume.Attach failed for volume "pvc-b40c11ba-9969-4878-b95e-9da79741227e" : timed out waiting for external-attacher of disk.csi.azure.com CSI driver to attach volume /subscriptions/e659c16c-8ba9-41ab-98a0-d64b0237ba45/resourceGroups/MC_cicd-aks-6ei3v2ud-group_cicd-aks-6ei3v2ud_eastus/providers/Microsoft.Compute/disks/pvc-b40c11ba-9969-4878-b95e-9da79741227e Warning FailedAttachVolume 3m event-controller Pod damengdb-tkwqqh-dmdb-1: AttachVolume.Attach failed for volume "pvc-2f9622fc-54fa-4cff-8c0a-610dd25c54bf" : timed out waiting for external-attacher of disk.csi.azure.com CSI driver to attach volume /subscriptions/e659c16c-8ba9-41ab-98a0-d64b0237ba45/resourceGroups/MC_cicd-aks-6ei3v2ud-group_cicd-aks-6ei3v2ud_eastus/providers/Microsoft.Compute/disks/pvc-2f9622fc-54fa-4cff-8c0a-610dd25c54bf Warning FailedAttachVolume 3m event-controller Pod damengdb-tkwqqh-dmdb-1: AttachVolume.Attach failed for volume "pvc-6b44a8e4-04cc-4172-85cb-a234b561d2d3" : timed out waiting for external-attacher of disk.csi.azure.com CSI driver to attach volume /subscriptions/e659c16c-8ba9-41ab-98a0-d64b0237ba45/resourceGroups/MC_cicd-aks-6ei3v2ud-group_cicd-aks-6ei3v2ud_eastus/providers/Microsoft.Compute/disks/pvc-6b44a8e4-04cc-4172-85cb-a234b561d2d3 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:58 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:30 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 check pod status done check cluster status again check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 2/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 3/3 Sep 01,2025 11:51 UTC+0800 ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Running 3/3 Sep 01,2025 11:51 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-verticalscaling-x9tlt ns-fkhrb VerticalScaling damengdb-tkwqqh dmdb Succeed 3/3 Sep 01,2025 11:51 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-verticalscaling-x9tlt --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-verticalscaling-x9tlt patched `kbcli cluster delete-ops --name damengdb-tkwqqh-verticalscaling-x9tlt --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-verticalscaling-x9tlt deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running check node drain check node drain success `kill 1` Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:58 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:59 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-2 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover fullcpu check cluster status before cluster-failover-fullcpu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-tkwqqh" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-damengdb-tkwqqh.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-damengdb-tkwqqh created apply test-chaos-mesh-fullcpu-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-fullcpu-damengdb-tkwqqh.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:01 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-tkwqqh" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart damengdb-tkwqqh --auto-approve --force=true --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-restart-f5hg6 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-restart-f5hg6 -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-restart-f5hg6 ns-fkhrb Restart damengdb-tkwqqh dmdb,dmmonitor Creating -/- Sep 01,2025 12:05 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:06 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:07 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:05 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-restart-f5hg6 ns-fkhrb Restart damengdb-tkwqqh dmdb,dmmonitor Succeed 4/4 Sep 01,2025 12:05 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-restart-f5hg6 ns-fkhrb Restart damengdb-tkwqqh dmdb,dmmonitor Succeed 4/4 Sep 01,2025 12:05 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-restart-f5hg6 --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-restart-f5hg6 patched `kbcli cluster delete-ops --name damengdb-tkwqqh-restart-f5hg6 --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-restart-f5hg6 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale offline instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-tkwqqh-hscaleoffinstance- labels: app.kubernetes.io/instance: damengdb-tkwqqh app.kubernetes.io/managed-by: kubeblocks namespace: ns-fkhrb spec: type: HorizontalScaling clusterName: damengdb-tkwqqh force: true horizontalScaling: - componentName: dmdb scaleIn: onlineInstancesToOffline: - damengdb-tkwqqh-dmdb-1 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-tkwqqh.yaml` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-hscaleoffinstance-t5ggw created create test_ops_cluster_damengdb-tkwqqh.yaml Success `rm -rf test_ops_cluster_damengdb-tkwqqh.yaml` check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-hscaleoffinstance-t5ggw ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Running 0/1 Sep 01,2025 12:07 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:06 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:05 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-hscaleoffinstance-t5ggw ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:07 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-hscaleoffinstance-t5ggw ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:07 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-hscaleoffinstance-t5ggw --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-hscaleoffinstance-t5ggw patched `kbcli cluster delete-ops --name damengdb-tkwqqh-hscaleoffinstance-t5ggw --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-hscaleoffinstance-t5ggw deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-2 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale online instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-tkwqqh-hscaleoninstance- labels: app.kubernetes.io/instance: damengdb-tkwqqh app.kubernetes.io/managed-by: kubeblocks namespace: ns-fkhrb spec: type: HorizontalScaling clusterName: damengdb-tkwqqh force: true horizontalScaling: - componentName: dmdb scaleOut: offlineInstancesToOnline: - damengdb-tkwqqh-dmdb-1 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-tkwqqh.yaml` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-hscaleoninstance-7fk49 created create test_ops_cluster_damengdb-tkwqqh.yaml Success `rm -rf test_ops_cluster_damengdb-tkwqqh.yaml` check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-hscaleoninstance-7fk49 ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Running 0/1 Sep 01,2025 12:08 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:06 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:10 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:05 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-hscaleoninstance-7fk49 ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:08 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-hscaleoninstance-7fk49 ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-hscaleoninstance-7fk49 --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-hscaleoninstance-7fk49 patched `kbcli cluster delete-ops --name damengdb-tkwqqh-hscaleoninstance-7fk49 --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-hscaleoninstance-7fk49 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop damengdb-tkwqqh --auto-approve --force=true --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-stop-pnr8p created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-stop-pnr8p -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-stop-pnr8p ns-fkhrb Stop damengdb-tkwqqh Running -/- Sep 01,2025 12:12 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Stopping Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-stop-pnr8p ns-fkhrb Stop damengdb-tkwqqh dmdb,dmmonitor Succeed 4/4 Sep 01,2025 12:12 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-stop-pnr8p ns-fkhrb Stop damengdb-tkwqqh dmdb,dmmonitor Succeed 4/4 Sep 01,2025 12:12 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-stop-pnr8p --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-stop-pnr8p patched `kbcli cluster delete-ops --name damengdb-tkwqqh-stop-pnr8p --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-stop-pnr8p deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start damengdb-tkwqqh --force=true --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-start-9wlv9 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-start-9wlv9 -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-start-9wlv9 ns-fkhrb Start damengdb-tkwqqh dmdb,dmmonitor Running 0/4 Sep 01,2025 12:12 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:13 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:14 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-start-9wlv9 ns-fkhrb Start damengdb-tkwqqh dmdb,dmmonitor Succeed 4/4 Sep 01,2025 12:12 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-start-9wlv9 ns-fkhrb Start damengdb-tkwqqh dmdb,dmmonitor Succeed 4/4 Sep 01,2025 12:12 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-start-9wlv9 --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-start-9wlv9 patched `kbcli cluster delete-ops --name damengdb-tkwqqh-start-9wlv9 --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-start-9wlv9 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 action: partition mode: all target: mode: all selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-damengdb-tkwqqh created apply test-chaos-mesh-networkpartition-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networkpartition-damengdb-tkwqqh.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:13 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:14 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-tkwqqh" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-tkwqqh namespace. `kbcli cluster hscale damengdb-tkwqqh --auto-approve --force=true --components dmdb --replicas 4 --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-horizontalscaling-z8d45 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-horizontalscaling-z8d45 -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-horizontalscaling-z8d45 ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Running -/- Sep 01,2025 12:17 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:13 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:14 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-3 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:22 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 damengdb-tkwqqh-dmdb-3 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-tkwqqh namespace. check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-horizontalscaling-z8d45 ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:17 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-horizontalscaling-z8d45 ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:17 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-horizontalscaling-z8d45 --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-horizontalscaling-z8d45 patched `kbcli cluster delete-ops --name damengdb-tkwqqh-horizontalscaling-z8d45 --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-horizontalscaling-z8d45 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data retry times: 1 check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-tkwqqh namespace. `kbcli cluster hscale damengdb-tkwqqh --auto-approve --force=true --components dmdb --replicas 3 --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-horizontalscaling-rqszg created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-horizontalscaling-rqszg -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-horizontalscaling-rqszg ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Running -/- Sep 01,2025 12:25 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:13 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:14 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-tkwqqh namespace. check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-horizontalscaling-rqszg ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:25 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-horizontalscaling-rqszg ns-fkhrb HorizontalScaling damengdb-tkwqqh dmdb Succeed 1/1 Sep 01,2025 12:25 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-horizontalscaling-rqszg --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-horizontalscaling-rqszg patched `kbcli cluster delete-ops --name damengdb-tkwqqh-horizontalscaling-rqszg --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-horizontalscaling-rqszg deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-tkwqqh" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-damengdb-tkwqqh.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-damengdb-tkwqqh created apply test-chaos-mesh-timeoffset-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-timeoffset-damengdb-tkwqqh.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:13 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:14 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-tkwqqh" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get dmdb standby pod name... get pod name timeout check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data retry times: 1 check readonly db_client batch data retry times: 2 check readonly db_client batch data retry times: 3 check readonly db_client batch data retry times: 4 check readonly db_client batch data retry times: 5 error: pod, type/name or --filename must be specified [Error] check readonly db_client batch data timeout test failover dnsrandom check cluster status before cluster-failover-dnsrandom cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "damengdb-tkwqqh-dmdb-1" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "damengdb-tkwqqh-dmdb-1" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-tkwqqh" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-damengdb-tkwqqh.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-damengdb-tkwqqh created apply test-chaos-mesh-dnsrandom-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-dnsrandom-damengdb-tkwqqh.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:32 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-tkwqqh" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-0 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-tkwqqh" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-damengdb-tkwqqh --namespace ns-fkhrb ` Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-tkwqqh" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-tkwqqh" not found `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-damengdb-tkwqqh namespace: ns-fkhrb spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local" - "--user" - "sysdba" - "--password" - "ZUAH8w9u6i" - "--port" - "5236" - "--database" - "sysdba" - "--dbtype" - "damengdb" - "--test" - "connectionstress" - "--connections" - "1500" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-damengdb-tkwqqh.yaml` pod/test-db-client-connectionstress-damengdb-tkwqqh created apply test-db-client-connectionstress-damengdb-tkwqqh.yaml Success `rm -rf test-db-client-connectionstress-damengdb-tkwqqh.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-damengdb-tkwqqh 1/1 Running 0 5s check pod test-db-client-connectionstress-damengdb-tkwqqh status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-damengdb-tkwqqh 0/1 Completed 0 9s check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:32 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done --host damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local --user sysdba --password ZUAH8w9u6i --port 5236 --database sysdba --dbtype damengdb --test connectionstress --connections 1500 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Test Result: Created 1500 connections Connection Information: Database Type: damengdb Host: damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local Port: 5236 Database: sysdba Table: User: sysdba Org: Access Mode: mysql Test Type: connectionstress Connection Count: 1500 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-damengdb-tkwqqh --namespace ns-fkhrb ` pod/test-db-client-connectionstress-damengdb-tkwqqh patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-damengdb-tkwqqh" force deleted check failover pod name failover pod name:damengdb-tkwqqh-dmdb-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=damengdb-tkwqqh,apps.kubeblocks.io/component-name=dmdb,apps.kubeblocks.io/vct-name=data --namespace ns-fkhrb ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-tkwqqh namespace. `kbcli cluster volume-expand damengdb-tkwqqh --auto-approve --force=true --components dmdb --volume-claim-templates data --storage 24Gi --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-volumeexpansion-4dtkx created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-volumeexpansion-4dtkx -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-volumeexpansion-4dtkx ns-fkhrb VolumeExpansion damengdb-tkwqqh dmdb Running -/- Sep 01,2025 12:37 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:32 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-tkwqqh namespace. check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-volumeexpansion-4dtkx ns-fkhrb VolumeExpansion damengdb-tkwqqh dmdb Succeed 3/3 Sep 01,2025 12:37 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-volumeexpansion-4dtkx ns-fkhrb VolumeExpansion damengdb-tkwqqh dmdb Succeed 3/3 Sep 01,2025 12:37 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-volumeexpansion-4dtkx --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-volumeexpansion-4dtkx patched `kbcli cluster delete-ops --name damengdb-tkwqqh-volumeexpansion-4dtkx --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-volumeexpansion-4dtkx deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-damengdb-tkwqqh.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-tkwqqh created apply test-chaos-mesh-oom-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-oom-damengdb-tkwqqh.yaml` check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:32 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-tkwqqh patched check failover pod name failover pod name:damengdb-tkwqqh-dmdb-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-damengdb-tkwqqh.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-tkwqqh created apply test-chaos-mesh-oom-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-oom-damengdb-tkwqqh.yaml` check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:32 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-tkwqqh" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-tkwqqh patched check failover pod name:damengdb-tkwqqh-dmdb-1 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-tkwqqh" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-damengdb-tkwqqh.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-damengdb-tkwqqh created apply test-chaos-mesh-podfailure-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-podfailure-damengdb-tkwqqh.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:47 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-tkwqqh" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster update monitor cluster does not need to check monitor currently check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:47 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-tkwqqh" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-damengdb-tkwqqh.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-damengdb-tkwqqh created apply test-chaos-mesh-networklossover-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-networklossover-damengdb-tkwqqh.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:47 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role No resources found in ns-fkhrb namespace. primary: ;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-tkwqqh" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-damengdb-tkwqqh patched check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-tkwqqh" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-tkwqqh" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-damengdb-tkwqqh namespace: ns-fkhrb spec: selector: namespaces: - ns-fkhrb labelSelectors: apps.kubeblocks.io/pod-name: damengdb-tkwqqh-dmdb-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-damengdb-tkwqqh.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-damengdb-tkwqqh created apply test-chaos-mesh-dnserror-damengdb-tkwqqh.yaml Success `rm -rf test-chaos-mesh-dnserror-damengdb-tkwqqh.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:47 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:33 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-1;standby: damengdb-tkwqqh-dmdb-0 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-damengdb-tkwqqh --namespace ns-fkhrb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-tkwqqh" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-tkwqqh" not found check failover pod name failover pod name:damengdb-tkwqqh-dmdb-1 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster configure component_tmp: dmdb check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure damengdb-tkwqqh --auto-approve --force=true --set BUFFER=900 --components dmdb --config-spec dmdb-config --config-file dm.ini --namespace ns-fkhrb ` Will updated configure file meta: ConfigSpec: dmdb-config ConfigFile: dm.ini ComponentName: dmdb ClusterName: damengdb-tkwqqh OpsRequest damengdb-tkwqqh-reconfiguring-g9x7l created successfully, you can view the progress: kbcli cluster describe-ops damengdb-tkwqqh-reconfiguring-g9x7l -n ns-fkhrb check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-reconfiguring-g9x7l ns-fkhrb Reconfiguring damengdb-tkwqqh dmdb Creating -/- Sep 01,2025 12:54 UTC+0800 check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:54 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-tkwqqh --status all --namespace ns-fkhrb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-tkwqqh-reconfiguring-g9x7l ns-fkhrb Reconfiguring damengdb-tkwqqh dmdb Succeed 2/2 Sep 01,2025 12:54 UTC+0800 check ops status done ops_status:damengdb-tkwqqh-reconfiguring-g9x7l ns-fkhrb Reconfiguring damengdb-tkwqqh dmdb Succeed 2/2 Sep 01,2025 12:54 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-tkwqqh-reconfiguring-g9x7l --namespace ns-fkhrb ` opsrequest.apps.kubeblocks.io/damengdb-tkwqqh-reconfiguring-g9x7l patched `kbcli cluster delete-ops --name damengdb-tkwqqh-reconfiguring-g9x7l --force --auto-approve --namespace ns-fkhrb ` OpsRequest damengdb-tkwqqh-reconfiguring-g9x7l deleted check config variables Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 900 configure:[BUFFER] result actual:[900] equal expected:[900] `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-ro.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-1 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update damengdb-tkwqqh --termination-policy=WipeOut --namespace ns-fkhrb ` cluster.apps.kubeblocks.io/damengdb-tkwqqh updated check cluster status `kbcli cluster list damengdb-tkwqqh --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh ns-fkhrb damengdb WipeOut Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=damengdb-tkwqqh,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-dmdb-0 ns-fkhrb damengdb-tkwqqh dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-1 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:55 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmdb-2 ns-fkhrb damengdb-tkwqqh dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:54 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-dmmonitor-0 ns-fkhrb damengdb-tkwqqh dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:12 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-tkwqqh-dmdb-0;standby: damengdb-tkwqqh-dmdb-1 damengdb-tkwqqh-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-tkwqqh` set secret: damengdb-tkwqqh-dmdb-account-sysdba `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-tkwqqh-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:ZUAH8w9u6i;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-fkhrb namespace. check cluster connect `echo " ./disql sysdba/ZUAH8w9u6i@damengdb-tkwqqh-dmdb-rw.ns-fkhrb.svc.cluster.local:5236 " | kubectl exec -it damengdb-tkwqqh-dmdb-0 --namespace ns-fkhrb -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done cluster full backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-bxwnt -n kb-hmwej -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-bxwnt -n kb-hmwej -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-hmwej get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-hmwej -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-hmwej -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-hmwej.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-hmwej.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-669d87677b-7sjlg --namespace kb-hmwej -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup damengdb-tkwqqh --method full --namespace ns-fkhrb ` Backup backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 -n ns-fkhrb check backup status `kbcli cluster list-backups damengdb-tkwqqh --namespace ns-fkhrb ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 ns-fkhrb damengdb-tkwqqh full Sep 01,2025 12:56 UTC+0800 backup_status:damengdb-tkwqqh-full-Running backup_status:damengdb-tkwqqh-full-Running backup_status:damengdb-tkwqqh-full-Running backup_status:damengdb-tkwqqh-full-Running check backup status done backup_status:backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 ns-fkhrb damengdb-tkwqqh full Completed 1036723 16s Sep 01,2025 12:56 UTC+0800 Sep 01,2025 12:56 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "damengdb-tkwqqh-backup" not found `kbcli cluster describe-backup backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 --namespace ns-fkhrb ` Name: backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 Cluster: damengdb-tkwqqh Namespace: ns-fkhrb Spec: Method: full Policy Name: damengdb-tkwqqh-dmdb-backup-policy Status: Phase: Completed Total Size: 1036723 ActionSet Name: dmdb-full-backup Repository: backuprepo-kbcli-test Duration: 16s Start Time: Sep 01,2025 12:56 UTC+0800 Completion Time: Sep 01,2025 12:56 UTC+0800 Path: /ns-fkhrb/damengdb-tkwqqh-c1f73216-28ad-4a1c-b972-dee7b6f5fe86/dmdb/backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 Time Range Start: Sep 01,2025 12:55 UTC+0800 Time Range End: Sep 01,2025 12:56 UTC+0800 Warning Events: `kbcli cluster restore damengdb-tkwqqh-backup --backup backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 --namespace ns-fkhrb ` Cluster damengdb-tkwqqh-backup created check cluster status `kbcli cluster list damengdb-tkwqqh-backup --show-labels --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-tkwqqh-backup ns-fkhrb damengdb WipeOut Sep 01,2025 12:57 UTC+0800 clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster damengdb-tkwqqh-backup yaml-------------------------------------- `kubectl get cluster damengdb-tkwqqh-backup -o yaml --namespace ns-fkhrb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"damengdb-tkwqqh-backup","type":"Restore"***]' kubeblocks.io/reconcile: "2025-09-01T04:51:05.279277521Z" kubeblocks.io/restore-from-backup: '***"dmdb":***"doReadyRestoreAfterClusterRunning":"false","name":"backup-ns-fkhrb-damengdb-tkwqqh-20250901125638","namespace":"ns-fkhrb","volumeRestorePolicy":"Parallel"***' creationTimestamp: "2025-09-01T04:57:00Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 1 labels: clusterdefinition.kubeblocks.io/name: damengdb clusterversion.kubeblocks.io/name: "" name: damengdb-tkwqqh-backup namespace: ns-fkhrb resourceVersion: "101330" uid: c7c31e3e-7d08-457d-a2fb-f41279ebb4b2 spec: clusterDefinitionRef: damengdb componentSpecs: - componentDef: dmdb-8.1 env: - name: OGUID value: "1" - name: UNICODE_FLAG value: "1" - name: LENGTH_IN_CHAR value: "1" - name: CASE_SENSITIVE value: "0" - name: BLANK_PAD_MODE value: "0" - name: ARCH_WAIT_APPLY value: "0" - name: HOST_NETWORK value: disabled - name: TIME_ZONE value: "+08:00" - name: ENABLE_AUDIT value: "true" - name: INSTANCE_NAME_PREFIX - name: PAGE_SIZE value: "8" - name: EXTENT_SIZE value: "16" - name: LOG_SIZE value: "2048" - name: SORT_MODE value: "0" - name: PK_CLUSTER_MODE value: "0" - name: ARCH_SPACE_LIMIT value: "51200" - name: IS_REMOTE_BACKUP value: "false" - name: REMOTE_TIMER_INTERVAL value: "1" name: dmdb podUpdatePolicy: Recreate replicas: 3 resources: limits: cpu: 1100m memory: 2254857830400m requests: cpu: 1100m memory: 2254857830400m serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 24Gi - name: arch spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumes: - name: license secret: secretName: damengdb-tkwqqh-license - componentDef: dmmonitor-8.1 env: - name: OGUID value: "1" name: dmmonitor podUpdatePolicy: Recreate replicas: 1 resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: WipeOut topology: realtime-replication status: clusterDefGeneration: 2 components: dmdb: phase: Creating podsReady: false dmmonitor: phase: Running podsReady: true podsReadyTime: "2025-09-01T04:57:19Z" conditions: - lastTransitionTime: "2025-09-01T04:57:00Z" message: 'The operator has started the provisioning of Cluster: damengdb-tkwqqh-backup' observedGeneration: 1 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T04:57:00Z" message: Successfully applied for resources observedGeneration: 1 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T04:57:00Z" message: 'pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T04:57:00Z" message: 'pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 1 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster damengdb-tkwqqh-backup-------------------------------------- `kubectl describe cluster damengdb-tkwqqh-backup --namespace ns-fkhrb ` Name: damengdb-tkwqqh-backup Namespace: ns-fkhrb Labels: clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"damengdb-tkwqqh-backup","type":"Restore"***] kubeblocks.io/reconcile: 2025-09-01T04:51:05.279277521Z kubeblocks.io/restore-from-backup: ***"dmdb":***"doReadyRestoreAfterClusterRunning":"false","name":"backup-ns-fkhrb-damengdb-tkwqqh-20250901125638","namespace":"ns-fkhrb","volum... API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T04:57:00Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Resource Version: 101330 UID: c7c31e3e-7d08-457d-a2fb-f41279ebb4b2 Spec: Cluster Definition Ref: damengdb Component Specs: Component Def: dmdb-8.1 Env: Name: OGUID Value: 1 Name: UNICODE_FLAG Value: 1 Name: LENGTH_IN_CHAR Value: 1 Name: CASE_SENSITIVE Value: 0 Name: BLANK_PAD_MODE Value: 0 Name: ARCH_WAIT_APPLY Value: 0 Name: HOST_NETWORK Value: disabled Name: TIME_ZONE Value: +08:00 Name: ENABLE_AUDIT Value: true Name: INSTANCE_NAME_PREFIX Name: PAGE_SIZE Value: 8 Name: EXTENT_SIZE Value: 16 Name: LOG_SIZE Value: 2048 Name: SORT_MODE Value: 0 Name: PK_CLUSTER_MODE Value: 0 Name: ARCH_SPACE_LIMIT Value: 51200 Name: IS_REMOTE_BACKUP Value: false Name: REMOTE_TIMER_INTERVAL Value: 1 Name: dmdb Pod Update Policy: Recreate Replicas: 3 Resources: Limits: Cpu: 1100m Memory: 2254857830400m Requests: Cpu: 1100m Memory: 2254857830400m Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 24Gi Name: arch Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Volumes: Name: license Secret: Secret Name: damengdb-tkwqqh-license Component Def: dmmonitor-8.1 Env: Name: OGUID Value: 1 Name: dmmonitor Pod Update Policy: Recreate Replicas: 1 Resources: Limits: Cpu: 1 Memory: 2Gi Requests: Cpu: 1 Memory: 2Gi Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: WipeOut Topology: realtime-replication Status: Cluster Def Generation: 2 Components: Dmdb: Phase: Creating Pods Ready: false Dmmonitor: Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T04:57:19Z Conditions: Last Transition Time: 2025-09-01T04:57:00Z Message: The operator has started the provisioning of Cluster: damengdb-tkwqqh-backup Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T04:57:00Z Message: Successfully applied for resources Observed Generation: 1 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T04:57:00Z Message: pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T04:57:00Z Message: pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 1 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 7m26s cluster-controller The operator has started the provisioning of Cluster: damengdb-tkwqqh-backup Normal ApplyResourcesSucceed 7m26s cluster-controller Successfully applied for resources Normal ComponentPhaseTransition 7m7s cluster-controller component is Running Warning ReplicasNotReady 7m7s cluster-controller pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Warning ComponentsNotReady 7m7s cluster-controller pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Normal NeedWaiting 7m4s (x9 over 7m26s) component-controller waiting for restore "damengdb-tkwqqh-backup-dmdb-c7c31e3e-preparedata" successfully Normal ComponentPhaseTransition 7m1s (x2 over 7m25s) cluster-controller component is Creating ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances damengdb-tkwqqh-backup --namespace ns-fkhrb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-tkwqqh-backup-dmdb-0 ns-fkhrb damengdb-tkwqqh-backup dmdb Running 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:57 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-backup-dmdb-1 ns-fkhrb damengdb-tkwqqh-backup dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:58 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-backup-dmdb-2 ns-fkhrb damengdb-tkwqqh-backup dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:24Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:58 UTC+0800 arch:20Gi log:20Gi damengdb-tkwqqh-backup-dmmonitor-0 ns-fkhrb damengdb-tkwqqh-backup dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:57 UTC+0800 check pod status done check cluster status again `kbcli cluster describe-backup backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 --namespace ns-fkhrb ` Name: backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 Cluster: damengdb-tkwqqh Namespace: ns-fkhrb Spec: Method: full Policy Name: damengdb-tkwqqh-dmdb-backup-policy Status: Phase: Completed Total Size: 1036723 ActionSet Name: dmdb-full-backup Repository: backuprepo-kbcli-test Duration: 16s Start Time: Sep 01,2025 12:56 UTC+0800 Completion Time: Sep 01,2025 12:56 UTC+0800 Path: /ns-fkhrb/damengdb-tkwqqh-c1f73216-28ad-4a1c-b972-dee7b6f5fe86/dmdb/backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 Time Range Start: Sep 01,2025 12:55 UTC+0800 Time Range End: Sep 01,2025 12:56 UTC+0800 Warning Events: delete cluster damengdb-tkwqqh-backup `kbcli cluster delete damengdb-tkwqqh-backup --auto-approve --namespace ns-fkhrb ` Cluster damengdb-tkwqqh-backup deleted Error from server (NotFound): secrets "damengdb-tkwqqh-backup-license" not found Error from server (NotFound): secrets "damengdb-tkwqqh-backup-license" not found Error from server (NotFound): secrets "damengdb-tkwqqh-backup-license" not found pod_info:damengdb-tkwqqh-backup-dmdb-0 4/4 Terminating 0 7m8s damengdb-tkwqqh-backup-dmdb-1 4/4 Terminating 0 6m29s damengdb-tkwqqh-backup-dmdb-2 4/4 Terminating 0 5m52s damengdb-tkwqqh-backup-dmmonitor-0 1/1 Terminating 0 7m33s No resources found in ns-fkhrb namespace. delete cluster pod done No resources found in ns-fkhrb namespace. check cluster resource non-exist OK: pvc No resources found in ns-fkhrb namespace. delete cluster done No resources found in ns-fkhrb namespace. No resources found in ns-fkhrb namespace. No resources found in ns-fkhrb namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 --namespace ns-fkhrb ` backup.dataprotection.kubeblocks.io/backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 patched `kbcli cluster delete-backup damengdb-tkwqqh --name backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 --force --auto-approve --namespace ns-fkhrb ` Backup backup-ns-fkhrb-damengdb-tkwqqh-20250901125638 deleted No opsrequests found in ns-fkhrb namespace. cluster list-logs `kbcli cluster list-logs damengdb-tkwqqh --namespace ns-fkhrb ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update damengdb-tkwqqh --enable-all-logs=true --namespace ns-fkhrb Error from server (NotFound): pods "damengdb-tkwqqh-dmdb-0" not found cluster logs `kbcli cluster logs damengdb-tkwqqh --tail 30 --namespace ns-fkhrb ` Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) 2025-09-01 04:56:41.658 [INFO] database P0000000610 T0000000000000001138 buf4_wait_for_flush end clsn[123497] 2025-09-01 04:56:41.658 [INFO] database P0000000610 T0000000000000001138 buf4_wait_for_flush begin flush_lsn[123497], timeout[30000] 2025-09-01 04:56:41.658 [INFO] database P0000000610 T0000000000000000664 checkpoint begin, used_space[40960], free_space[4294918144]... 2025-09-01 04:56:41.667 [INFO] database P0000000610 T0000000000000000664 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-09-01 04:56:41.667 [INFO] database P0000000610 T0000000000000000664 ckpt2_log_adjust: ckpt_lsn(123649), ckpt_fil(0), ckpt_off(62971904), cur_lsn(123676), l_next_seq(16340), g_next_seq(16303), cur_free(63016960), total_space(4294959104), used_space(45056), free_space(4294914048), n_ep(1) 2025-09-01 04:56:41.667 [INFO] database P0000000610 T0000000000000000664 checkpoint end, 0 pages flushed, used_space[45056], free_space[4294914048]. 2025-09-01 04:56:41.667 [INFO] database P0000000610 T0000000000000001138 buf4_wait_for_flush end clsn[123649] 2025-09-01 04:56:41.667 [INFO] database P0000000610 T0000000000000001138 hpc_begin_apply_info_get_online p_db_magic(0), n_apply_ep(0), pkg_seq_arr([16292, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), lsn_arr([123495, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 2025-09-01 04:56:41.667 [INFO] database P0000000610 T0000000000000001138 rfil[/opt/dmdbms/data/DAMENG/DAMENG01.log], first local_rpkg_seqno = 16330, global_rpkg_seqno = 16293, first_startup = 0 2025-09-01 04:56:41.685 [INFO] database P0000000610 T0000000000000001138 hpc_begin_apply_info_get_online p_db_magic(0), n_apply_ep(1), pkg_seq_arr([16293, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), lsn_arr([123649, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 2025-09-01 04:56:47.170 [INFO] database P0000000610 T0000000000000001138 bak2_dbf_backup_exec finished, code:0, sess[0x7f03dc011498], memobj[0x7f03dc0114a0] 2025-09-01 04:56:47.189 [INFO] database P0000000610 T0000000000000001138 socket_err_should_retry errno:104 2025-09-01 04:56:47.854 [INFO] database P0000000610 T0000000000000000658 rafil_close_with_trunc file[/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x7A4680_EP0_2025-09-01_04-55-39.log] to 61440 len, g_next_seq:16305, ret = 1! 2025-09-01 04:56:47.854 [INFO] database P0000000610 T0000000000000000658 rfil_close_low set arch rfil[/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x7A4680_EP0_2025-09-01_04-55-39.log]'s sta to inactive, l_next_seq = 16305, g_next_seq = 16305, clsn = 123680, handle = 12, free=61440, len=61440 2025-09-01 04:59:41.031 [INFO] database P0000000610 T0000000000000000697 checkpoint requested by CKPT_INTERVAL, rlog free space[4294901760], used space[57344] 2025-09-01 04:59:41.031 [INFO] database P0000000610 T0000000000000000697 checkpoint generate by ckpt_interval 2025-09-01 04:59:41.031 [INFO] database P0000000610 T0000000000000000664 checkpoint begin, used_space[57344], free_space[4294901760]... 2025-09-01 04:59:41.036 [INFO] database P0000000610 T0000000000000000658 rafil_file_create [/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x7A4680_EP0_2025-09-01_04-59-41.log] start 2025-09-01 04:59:41.039 [INFO] database P0000000610 T0000000000000000658 rfil_close_low set arch rfil[/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x7A4680_EP0_2025-09-01_04-59-41.log]'s sta to inactive, l_next_seq = 2457, g_next_seq = 2457, clsn = 0, handle = 12, free=4096, len=134217728 2025-09-01 04:59:41.043 [INFO] database P0000000610 T0000000000000000658 rafil_file_create [/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x7A4680_EP0_2025-09-01_04-59-41.log] success, handle = 12 2025-09-01 04:59:41.043 [INFO] database P0000000610 T0000000000000000658 rafil_file_init, rac_seqno:0, create new file and set last_arch_fil:/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x7A4680_EP0_2025-09-01_04-59-41.log! 2025-09-01 04:59:41.048 [INFO] database P0000000610 T0000000000000000664 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-09-01 04:59:41.048 [INFO] database P0000000610 T0000000000000000664 ckpt2_log_adjust: ckpt_lsn(123652), ckpt_fil(0), ckpt_off(62980096), cur_lsn(123680), l_next_seq(16344), g_next_seq(16307), cur_free(63033344), total_space(4294959104), used_space(53248), free_space(4294905856), n_ep(1) 2025-09-01 04:59:41.048 [INFO] database P0000000610 T0000000000000000664 checkpoint end, 0 pages flushed, used_space[53248], free_space[4294905856]. 2025-09-01 05:02:41.071 [INFO] database P0000000610 T0000000000000000697 checkpoint requested by CKPT_INTERVAL, rlog free space[4294905856], used space[53248] 2025-09-01 05:02:41.071 [INFO] database P0000000610 T0000000000000000697 checkpoint generate by ckpt_interval 2025-09-01 05:02:41.071 [INFO] database P0000000610 T0000000000000000664 checkpoint begin, used_space[53248], free_space[4294905856]... 2025-09-01 05:02:41.080 [INFO] database P0000000610 T0000000000000000664 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-09-01 05:02:41.080 [INFO] database P0000000610 T0000000000000000664 ckpt2_log_adjust: ckpt_lsn(123653), ckpt_fil(0), ckpt_off(62980096), cur_lsn(123680), l_next_seq(16345), g_next_seq(16308), cur_free(63037440), total_space(4294959104), used_space(57344), free_space(4294901760), n_ep(1) 2025-09-01 05:02:41.081 [INFO] database P0000000610 T0000000000000000664 checkpoint end, 0 pages flushed, used_space[57344], free_space[4294901760]. delete cluster damengdb-tkwqqh `kbcli cluster delete damengdb-tkwqqh --auto-approve --namespace ns-fkhrb ` Cluster damengdb-tkwqqh deleted secret "damengdb-tkwqqh-license" deleted pod_info:damengdb-tkwqqh-dmdb-0 4/4 Running 0 9m55s damengdb-tkwqqh-dmdb-1 4/4 Running 0 9m18s damengdb-tkwqqh-dmdb-2 4/4 Running 0 10m damengdb-tkwqqh-dmmonitor-0 1/1 Running 0 52m No resources found in ns-fkhrb namespace. delete cluster pod done No resources found in ns-fkhrb namespace. check cluster resource non-exist OK: pvc No resources found in ns-fkhrb namespace. delete cluster done No resources found in ns-fkhrb namespace. No resources found in ns-fkhrb namespace. No resources found in ns-fkhrb namespace. Damengdb Test Suite All Done! --------------------------------------Damengdb (Topology = realtime-replication Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=realtime-replication;ComponentVersion=damengdb;ServiceVersion=8.1.3-26-20240821;]|[Description=Create a cluster with the specified topology realtime-replication and component version damengdb and service version 8.1.3-26-20240821] [PASSED]|[Connect]|[ComponentName=dmdb]|[Description=Connect to the cluster] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=dmdb]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [WARNING]|[Operation]|[Succeed Or Failed Soon]|[Description=-] [FAILED]|[SwitchOver]|[ComponentName=dmdb]|[Description=SwitchOver the cluster specify component dmdb] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=dmdb]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [SKIPPED]|[No-Failover]|[HA=Evicting Pod;ComponentName=dmdb]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=dmdb]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=dmdb]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[VerticalScaling]|[ComponentName=dmdb]|[Description=VerticalScaling the cluster specify component dmdb] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=dmdb]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[HscaleOfflineInstances]|[ComponentName=dmdb]|[Description=Hscale the cluster instances offline specify component dmdb] [PASSED]|[HscaleOnlineInstances]|[ComponentName=dmdb]|[Description=Hscale the cluster instances online specify component dmdb] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=dmdb]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[HorizontalScaling Out]|[ComponentName=dmdb]|[Description=HorizontalScaling Out the cluster specify component dmdb] [PASSED]|[HorizontalScaling In]|[ComponentName=dmdb]|[Description=HorizontalScaling In the cluster specify component dmdb] [FAILED]|[CheckDBClientBatchDataReadonly]|[OpsType=timeoffset;Role=Readonly]|[Description=Check the cluster readonly batch data consistent] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=dmdb]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=dmdb]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[VolumeExpansion]|[ComponentName=dmdb]|[Description=VolumeExpansion the cluster specify component dmdb] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=dmdb]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Reconfiguring]|[ComponentName=dmdb;BUFFER=900]|[Description=Reconfiguring the cluster specify component dmdb set BUFFER=900] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=full]|[Description=The cluster full Backup] [FAILED]|[Restore]|[BackupMethod=full]|[Description=The cluster full Restore] [PASSED]|[Delete Restore Cluster]|[BackupMethod=full]|[Description=Delete the full restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]