source commons files source engines files source kubeblocks files source kubedb files CLUSTER_NAME: `kubectl get namespace | grep ns-aihga ` `kubectl create namespace ns-aihga` namespace/ns-aihga created create namespace ns-aihga done check registry secret No resources found in ns-aihga namespace. create docker registry server https://index.docker.io/v1/ secret/kbcli-test-registry-key created create docker registry secret kbcli-test-registry-key Success check ns:ns-aihga sa:default imagePullSecrets patch ns:ns-aihga sa:default imagePullSecrets serviceaccount/default patched patch ns:ns-aihga sa:default imagePullSecrets Success download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 51.4M 0 --:--:-- --:--:-- --:--:-- 51.4M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.1 done Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Kubernetes Env: v1.32.6 check snapshot controller check snapshot controller done POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default KubeBlocks version is:1.0.1 skip upgrade KubeBlocks current KubeBlocks version: 1.0.1 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:dmdb set component version set component version:damengdb set service versions:8.1.4-6-20241231,8.1.3-26-20240821,8.1.3-100-20240124 set service versions sorted:8.1.3-100-20240124,8.1.3-26-20240821,8.1.4-6-20241231 set damengdb component definition set damengdb component definition dmdb-repl-8.1.3-1.0.1 REPORT_COUNT 0:0 set replicas first:2,8.1.3-100-20240124|2,8.1.3-26-20240821 set replicas third:2,8.1.3-26-20240821 set replicas fourth:2,8.1.3-100-20240124 set minimum cmpv service version set minimum cmpv service version replicas:2,8.1.3-26-20240821 REPORT_COUNT:1 CLUSTER_TOPOLOGY:realtime-replication topology realtime-replication found in cluster definition damengdb set damengdb component definition set damengdb component definition dmdb-repl-8.1.3-1.0.1 LIMIT_CPU:1 LIMIT_MEMORY:2 storage size: 20 CLUSTER_NAME:damengdb-mycwkx No resources found in ns-aihga namespace. pod_info: termination_policy:WipeOut create 2 replica WipeOut damengdb cluster check component definition set component definition by component version check cmpd by labels set component definition1: dmdb-repl-8.1.3-1.0.1 by component version:damengdb apiVersion: v1 kind: Secret metadata: name: damengdb-mycwkx-license namespace: ns-aihga type: Opaque --- apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: damengdb-mycwkx namespace: ns-aihga spec: clusterDef: damengdb topology: realtime-replication terminationPolicy: WipeOut componentSpecs: - name: dmdb serviceVersion: 8.1.3-26-20240821 replicas: 2 env: - name: "OGUID" value: "1" - name: "UNICODE_FLAG" value: "1" - name: "LENGTH_IN_CHAR" value: "1" - name: "CASE_SENSITIVE" value: "0" - name: "BLANK_PAD_MODE" value: "0" - name: "ARCH_WAIT_APPLY" value: "0" - name: "HOST_NETWORK" value: "disabled" - name: "TIME_ZONE" value: "+08:00" - name: "ENABLE_AUDIT" value: "true" - name: "INSTANCE_NAME_PREFIX" value: "" - name: "PAGE_SIZE" value: "8" - name: "EXTENT_SIZE" value: "16" - name: "LOG_SIZE" value: "2048" - name: "SORT_MODE" value: "0" - name: "PK_CLUSTER_MODE" value: "0" - name: "ARCH_SPACE_LIMIT" value: "51200" - name: "IS_REMOTE_BACKUP" value: "false" - name: "REMOTE_TIMER_INTERVAL" value: "1" volumes: - name: license secret: secretName: damengdb-mycwkx-license resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: arch spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: dmmonitor serviceVersion: 8.1.3-26-20240821 replicas: 1 env: - name: "OGUID" value: "1" - name: "SERVICE_VERSION" value: "8.1.3-26-20240821" resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi `kubectl apply -f test_create_damengdb-mycwkx.yaml` secret/damengdb-mycwkx-license created cluster.apps.kubeblocks.io/damengdb-mycwkx created apply test_create_damengdb-mycwkx.yaml Success `rm -rf test_create_damengdb-mycwkx.yaml` `kubectl get serviceaccount --no-headers --namespace ns-aihga -l app.kubernetes.io/instance=damengdb-mycwkx` patch ns:ns-aihga sa:kb-dmdb-repl-8.1.3-1.0.1 imagePullSecrets serviceaccount/kb-dmdb-repl-8.1.3-1.0.1 patched patch ns:ns-aihga sa:kb-dmmonitor-8.1-1.0.1 imagePullSecrets serviceaccount/kb-dmmonitor-8.1-1.0.1 patched patch ns:ns-aihga sa:kb-dmdb-repl-8.1.3-1.0.1 kb-dmmonitor-8.1-1.0.1 imagePullSecrets Success check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Creating Sep 12,2025 13:05 UTC+0800 clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:05 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:07 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check pod damengdb-mycwkx-dmdb-0 container_name dmdb exist password 75wHfEy7od check pod damengdb-mycwkx-dmdb-0 container_name exporter exist password 75wHfEy7od check pod damengdb-mycwkx-dmdb-0 container_name kbagent exist password 75wHfEy7od check pod damengdb-mycwkx-dmdb-0 container_name config-manager exist password 75wHfEy7od No container logs contain secret password. describe cluster `kbcli cluster describe damengdb-mycwkx --namespace ns-aihga ` Name: damengdb-mycwkx Created Time: Sep 12,2025 13:05 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-aihga damengdb realtime-replication Running WipeOut Endpoints: COMPONENT INTERNAL EXTERNAL dmdb damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME dmdb 8.1.3-26-20240821 damengdb-mycwkx-dmdb-0 primary Running 0 aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:05 UTC+0800 dmdb 8.1.3-26-20240821 damengdb-mycwkx-dmdb-1 standby Running 0 aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:07 UTC+0800 dmmonitor 8.1.3-26-20240821 damengdb-mycwkx-dmmonitor-0 Running 0 aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:05 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS dmdb 1 / 1 2Gi / 2Gi data:20Gi default arch:20Gi default log:20Gi default dmmonitor 1 / 1 2Gi / 2Gi data:20Gi default Images: COMPONENT COMPONENT-DEFINITION IMAGE dmdb dmdb-repl-8.1.3-1.0.1 docker.io/apecloud/dm:8.1.3-26 docker.io/apecloud/dmdb-exporter:0.1.1 docker.io/apecloud/dmdb-tool:8.1 dmmonitor dmmonitor-8.1-1.0.1 docker.io/apecloud/dm:8.1.3-26 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-aihga damengdb-mycwkx `kbcli cluster label damengdb-mycwkx app.kubernetes.io/instance- --namespace ns-aihga ` label "app.kubernetes.io/instance" not found. `kbcli cluster label damengdb-mycwkx app.kubernetes.io/instance=damengdb-mycwkx --namespace ns-aihga ` `kbcli cluster label damengdb-mycwkx --list --namespace ns-aihga ` NAME NAMESPACE LABELS damengdb-mycwkx ns-aihga app.kubernetes.io/instance=damengdb-mycwkx clusterdefinition.kubeblocks.io/name=damengdb label cluster app.kubernetes.io/instance=damengdb-mycwkx Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=damengdb-mycwkx --namespace ns-aihga ` `kbcli cluster label damengdb-mycwkx --list --namespace ns-aihga ` NAME NAMESPACE LABELS damengdb-mycwkx ns-aihga app.kubernetes.io/instance=damengdb-mycwkx case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=damengdb label cluster case.name=kbcli.test1 Success `kbcli cluster label damengdb-mycwkx case.name=kbcli.test2 --overwrite --namespace ns-aihga ` `kbcli cluster label damengdb-mycwkx --list --namespace ns-aihga ` NAME NAMESPACE LABELS damengdb-mycwkx ns-aihga app.kubernetes.io/instance=damengdb-mycwkx case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=damengdb label cluster case.name=kbcli.test2 Success `kbcli cluster label damengdb-mycwkx case.name- --namespace ns-aihga ` `kbcli cluster label damengdb-mycwkx --list --namespace ns-aihga ` NAME NAMESPACE LABELS damengdb-mycwkx ns-aihga app.kubernetes.io/instance=damengdb-mycwkx clusterdefinition.kubeblocks.io/name=damengdb delete cluster label case.name Success cluster connect `echo 'cat /opt/dmdbms/data/DAMENG/dm.ini | grep MAX_SESSIONS' | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash ` Defaulted container "dmdb" out of: dmdb, exporter, kbagent, config-manager, init-kbagent (init), kbagent-worker (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file MAX_SESSIONS = 1500 #Maximum number of concurrent sessions connect cluster Success set max_connections to 1500 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-damengdb-mycwkx" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-damengdb-mycwkx --namespace ns-aihga ` Error from server (NotFound): pods "test-db-client-executionloop-damengdb-mycwkx" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-damengdb-mycwkx" not found `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-damengdb-mycwkx namespace: ns-aihga spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local" - "--user" - "sysdba" - "--password" - "75wHfEy7od" - "--port" - "5236" - "--dbtype" - "damengdb" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-damengdb-mycwkx.yaml` pod/test-db-client-executionloop-damengdb-mycwkx created apply test-db-client-executionloop-damengdb-mycwkx.yaml Success `rm -rf test-db-client-executionloop-damengdb-mycwkx.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 9s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 14s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 1/1 Running 0 62s check pod test-db-client-executionloop-damengdb-mycwkx status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-mycwkx 0/1 Completed 0 67s check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:05 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:07 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done --host damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local --user sysdba --password 75wHfEy7od --port 5236 --dbtype damengdb --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Execution loop start: tableCount: 0 create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); Execution loop start:INSERT INTO executions_loop_table (value) VALUES ('executions_loop_test_1'); [ 1s ] executions total: 155 successful: 155 failed: 0 disconnect: 0 [ 2s ] executions total: 320 successful: 320 failed: 0 disconnect: 0 [ 3s ] executions total: 480 successful: 480 failed: 0 disconnect: 0 [ 4s ] executions total: 652 successful: 652 failed: 0 disconnect: 0 [ 5s ] executions total: 827 successful: 827 failed: 0 disconnect: 0 [ 6s ] executions total: 1004 successful: 1004 failed: 0 disconnect: 0 [ 7s ] executions total: 1178 successful: 1178 failed: 0 disconnect: 0 [ 8s ] executions total: 1354 successful: 1354 failed: 0 disconnect: 0 [ 9s ] executions total: 1529 successful: 1529 failed: 0 disconnect: 0 [ 10s ] executions total: 1704 successful: 1704 failed: 0 disconnect: 0 [ 11s ] executions total: 1877 successful: 1877 failed: 0 disconnect: 0 [ 12s ] executions total: 2047 successful: 2047 failed: 0 disconnect: 0 [ 13s ] executions total: 2222 successful: 2222 failed: 0 disconnect: 0 [ 14s ] executions total: 2399 successful: 2399 failed: 0 disconnect: 0 [ 15s ] executions total: 2576 successful: 2576 failed: 0 disconnect: 0 [ 16s ] executions total: 2754 successful: 2754 failed: 0 disconnect: 0 [ 17s ] executions total: 2932 successful: 2932 failed: 0 disconnect: 0 [ 18s ] executions total: 3111 successful: 3111 failed: 0 disconnect: 0 [ 19s ] executions total: 3293 successful: 3293 failed: 0 disconnect: 0 [ 20s ] executions total: 3473 successful: 3473 failed: 0 disconnect: 0 [ 21s ] executions total: 3654 successful: 3654 failed: 0 disconnect: 0 [ 22s ] executions total: 3835 successful: 3835 failed: 0 disconnect: 0 [ 23s ] executions total: 4017 successful: 4017 failed: 0 disconnect: 0 [ 24s ] executions total: 4197 successful: 4197 failed: 0 disconnect: 0 [ 25s ] executions total: 4271 successful: 4271 failed: 0 disconnect: 0 [ 26s ] executions total: 4440 successful: 4440 failed: 0 disconnect: 0 [ 27s ] executions total: 4618 successful: 4618 failed: 0 disconnect: 0 [ 28s ] executions total: 4794 successful: 4794 failed: 0 disconnect: 0 [ 29s ] executions total: 4970 successful: 4970 failed: 0 disconnect: 0 [ 30s ] executions total: 5145 successful: 5145 failed: 0 disconnect: 0 [ 31s ] executions total: 5326 successful: 5326 failed: 0 disconnect: 0 [ 32s ] executions total: 5501 successful: 5501 failed: 0 disconnect: 0 [ 33s ] executions total: 5677 successful: 5677 failed: 0 disconnect: 0 [ 34s ] executions total: 5853 successful: 5853 failed: 0 disconnect: 0 [ 35s ] executions total: 6025 successful: 6025 failed: 0 disconnect: 0 [ 36s ] executions total: 6202 successful: 6202 failed: 0 disconnect: 0 [ 37s ] executions total: 6380 successful: 6380 failed: 0 disconnect: 0 [ 38s ] executions total: 6551 successful: 6551 failed: 0 disconnect: 0 [ 39s ] executions total: 6729 successful: 6729 failed: 0 disconnect: 0 [ 40s ] executions total: 6898 successful: 6898 failed: 0 disconnect: 0 [ 41s ] executions total: 7071 successful: 7071 failed: 0 disconnect: 0 [ 42s ] executions total: 7244 successful: 7244 failed: 0 disconnect: 0 [ 43s ] executions total: 7418 successful: 7418 failed: 0 disconnect: 0 [ 44s ] executions total: 7575 successful: 7575 failed: 0 disconnect: 0 [ 45s ] executions total: 7703 successful: 7703 failed: 0 disconnect: 0 [ 46s ] executions total: 7869 successful: 7869 failed: 0 disconnect: 0 [ 47s ] executions total: 8039 successful: 8039 failed: 0 disconnect: 0 [ 48s ] executions total: 8212 successful: 8212 failed: 0 disconnect: 0 [ 49s ] executions total: 8384 successful: 8384 failed: 0 disconnect: 0 [ 50s ] executions total: 8560 successful: 8560 failed: 0 disconnect: 0 [ 51s ] executions total: 8735 successful: 8735 failed: 0 disconnect: 0 [ 52s ] executions total: 8914 successful: 8914 failed: 0 disconnect: 0 [ 53s ] executions total: 9093 successful: 9093 failed: 0 disconnect: 0 [ 54s ] executions total: 9269 successful: 9269 failed: 0 disconnect: 0 [ 60s ] executions total: 9356 successful: 9356 failed: 0 disconnect: 0 Test Result: Total Executions: 9356 Successful Executions: 9356 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: damengdb Host: damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local Port: 5236 Database: Table: User: sysdba Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 9356 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-damengdb-mycwkx --namespace ns-aihga ` pod/test-db-client-executionloop-damengdb-mycwkx patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-damengdb-mycwkx" force deleted test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-mycwkx" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-damengdb-mycwkx --namespace ns-aihga ` Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-mycwkx" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-mycwkx" not found `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-damengdb-mycwkx namespace: ns-aihga spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local" - "--user" - "sysdba" - "--password" - "75wHfEy7od" - "--port" - "5236" - "--database" - "sysdba" - "--dbtype" - "damengdb" - "--test" - "connectionstress" - "--connections" - "1500" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-damengdb-mycwkx.yaml` pod/test-db-client-connectionstress-damengdb-mycwkx created apply test-db-client-connectionstress-damengdb-mycwkx.yaml Success `rm -rf test-db-client-connectionstress-damengdb-mycwkx.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-damengdb-mycwkx 1/1 Running 0 5s check pod test-db-client-connectionstress-damengdb-mycwkx status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-damengdb-mycwkx 0/1 Completed 0 9s check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:05 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:07 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done --host damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local --user sysdba --password 75wHfEy7od --port 5236 --database sysdba --dbtype damengdb --test connectionstress --connections 1500 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Test Result: Created 1500 connections Connection Information: Database Type: damengdb Host: damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local Port: 5236 Database: sysdba Table: User: sysdba Org: Access Mode: mysql Test Type: connectionstress Connection Count: 1500 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-damengdb-mycwkx --namespace ns-aihga ` pod/test-db-client-connectionstress-damengdb-mycwkx patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-damengdb-mycwkx" force deleted check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=damengdb-mycwkx,apps.kubeblocks.io/component-name=dmdb,apps.kubeblocks.io/vct-name=data --namespace ns-aihga ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-mycwkx namespace. `kbcli cluster volume-expand damengdb-mycwkx --auto-approve --force=true --components dmdb --volume-claim-templates data --storage 25Gi --namespace ns-aihga ` OpsRequest damengdb-mycwkx-volumeexpansion-7xnb9 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-volumeexpansion-7xnb9 -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-volumeexpansion-7xnb9 ns-aihga VolumeExpansion damengdb-mycwkx dmdb Running 0/2 Sep 12,2025 13:11 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:05 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:07 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:05 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-mycwkx namespace. check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-volumeexpansion-7xnb9 ns-aihga VolumeExpansion damengdb-mycwkx dmdb Succeed 2/2 Sep 12,2025 13:11 UTC+0800 check ops status done ops_status:damengdb-mycwkx-volumeexpansion-7xnb9 ns-aihga VolumeExpansion damengdb-mycwkx dmdb Succeed 2/2 Sep 12,2025 13:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-volumeexpansion-7xnb9 --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-volumeexpansion-7xnb9 patched `kbcli cluster delete-ops --name damengdb-mycwkx-volumeexpansion-7xnb9 --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-volumeexpansion-7xnb9 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop damengdb-mycwkx --auto-approve --force=true --namespace ns-aihga ` OpsRequest damengdb-mycwkx-stop-vl2wk created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-stop-vl2wk -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-stop-vl2wk ns-aihga Stop damengdb-mycwkx dmdb,dmmonitor Running 0/3 Sep 12,2025 13:16 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Stopping Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-stop-vl2wk ns-aihga Stop damengdb-mycwkx dmdb,dmmonitor Succeed 3/3 Sep 12,2025 13:16 UTC+0800 check ops status done ops_status:damengdb-mycwkx-stop-vl2wk ns-aihga Stop damengdb-mycwkx dmdb,dmmonitor Succeed 3/3 Sep 12,2025 13:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-stop-vl2wk --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-stop-vl2wk patched `kbcli cluster delete-ops --name damengdb-mycwkx-stop-vl2wk --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-stop-vl2wk deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start damengdb-mycwkx --force=true --namespace ns-aihga ` OpsRequest damengdb-mycwkx-start-dcxbt created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-start-dcxbt -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-start-dcxbt ns-aihga Start damengdb-mycwkx dmdb,dmmonitor Running 0/3 Sep 12,2025 13:17 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-start-dcxbt ns-aihga Start damengdb-mycwkx dmdb,dmmonitor Succeed 3/3 Sep 12,2025 13:17 UTC+0800 check ops status done ops_status:damengdb-mycwkx-start-dcxbt ns-aihga Start damengdb-mycwkx dmdb,dmmonitor Succeed 3/3 Sep 12,2025 13:17 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-start-dcxbt --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-start-dcxbt patched `kbcli cluster delete-ops --name damengdb-mycwkx-start-dcxbt --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-start-dcxbt deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-mycwkx" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-damengdb-mycwkx.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-damengdb-mycwkx created apply test-chaos-mesh-networkcorruptover-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-damengdb-mycwkx.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0 damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0 damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-mycwkx" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-damengdb-mycwkx.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-mycwkx created apply test-chaos-mesh-oom-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-oom-damengdb-mycwkx.yaml` check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-mycwkx patched check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-damengdb-mycwkx.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-mycwkx created apply test-chaos-mesh-oom-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-oom-damengdb-mycwkx.yaml` check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-mycwkx" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-mycwkx patched check failover pod name:damengdb-mycwkx-dmdb-0 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-mycwkx" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-damengdb-mycwkx.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-damengdb-mycwkx created apply test-chaos-mesh-dnsrandom-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-dnsrandom-damengdb-mycwkx.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-mycwkx" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster does not need to check monitor currently check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done cluster dmdb scale-out cluster dmdb scale-out replicas: 3 check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-mycwkx namespace. `kbcli cluster scale-out damengdb-mycwkx --auto-approve --force=true --components dmdb --replicas 1 --namespace ns-aihga ` OpsRequest damengdb-mycwkx-horizontalscaling-v58dk created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-horizontalscaling-v58dk -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-horizontalscaling-v58dk ns-aihga HorizontalScaling damengdb-mycwkx dmdb Running 0/1 Sep 12,2025 13:25 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-2 ns-aihga damengdb-mycwkx dmdb Running not-open 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000001/10.224.0.7 Sep 12,2025 13:25 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-mycwkx namespace. check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-horizontalscaling-v58dk ns-aihga HorizontalScaling damengdb-mycwkx dmdb Succeed 1/1 Sep 12,2025 13:25 UTC+0800 check ops status done ops_status:damengdb-mycwkx-horizontalscaling-v58dk ns-aihga HorizontalScaling damengdb-mycwkx dmdb Succeed 1/1 Sep 12,2025 13:25 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-horizontalscaling-v58dk --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-horizontalscaling-v58dk patched `kbcli cluster delete-ops --name damengdb-mycwkx-horizontalscaling-v58dk --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-horizontalscaling-v58dk deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster dmdb scale-in cluster dmdb scale-in replicas: 2 check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-mycwkx namespace. `kbcli cluster scale-in damengdb-mycwkx --auto-approve --force=true --components dmdb --replicas 1 --namespace ns-aihga ` OpsRequest damengdb-mycwkx-horizontalscaling-tw474 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-horizontalscaling-tw474 -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-horizontalscaling-tw474 ns-aihga HorizontalScaling damengdb-mycwkx dmdb Running 0/1 Sep 12,2025 13:30 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-mycwkx namespace. check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-horizontalscaling-tw474 ns-aihga HorizontalScaling damengdb-mycwkx dmdb Succeed 1/1 Sep 12,2025 13:30 UTC+0800 check ops status done ops_status:damengdb-mycwkx-horizontalscaling-tw474 ns-aihga HorizontalScaling damengdb-mycwkx dmdb Succeed 1/1 Sep 12,2025 13:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-horizontalscaling-tw474 --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-horizontalscaling-tw474 patched `kbcli cluster delete-ops --name damengdb-mycwkx-horizontalscaling-tw474 --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-horizontalscaling-tw474 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-mycwkx" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-damengdb-mycwkx.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-damengdb-mycwkx created apply test-chaos-mesh-timeoffset-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-timeoffset-damengdb-mycwkx.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-mycwkx" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running `kill 1` Defaulted container "dmdb" out of: dmdb, exporter, kbagent, config-manager, init-kbagent (init), kbagent-worker (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Abnormal Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Abnormal cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-mycwkx" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-damengdb-mycwkx.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-damengdb-mycwkx created apply test-chaos-mesh-networklossover-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-networklossover-damengdb-mycwkx.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:17 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1 / 1 2Gi / 2Gi data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:18 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0;standby: check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-mycwkx" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale damengdb-mycwkx --auto-approve --force=true --components dmdb --cpu 1100m --memory 2.1Gi --namespace ns-aihga ` OpsRequest damengdb-mycwkx-verticalscaling-4p5dn created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-verticalscaling-4p5dn -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-verticalscaling-4p5dn ns-aihga VerticalScaling damengdb-mycwkx dmdb Running 0/2 Sep 12,2025 13:36 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:37 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:36 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-verticalscaling-4p5dn ns-aihga VerticalScaling damengdb-mycwkx dmdb Succeed 2/2 Sep 12,2025 13:36 UTC+0800 check ops status done ops_status:damengdb-mycwkx-verticalscaling-4p5dn ns-aihga VerticalScaling damengdb-mycwkx dmdb Succeed 2/2 Sep 12,2025 13:36 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-verticalscaling-4p5dn --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-verticalscaling-4p5dn patched `kbcli cluster delete-ops --name damengdb-mycwkx-verticalscaling-4p5dn --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-verticalscaling-4p5dn deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-mycwkx" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-damengdb-mycwkx.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-damengdb-mycwkx created apply test-chaos-mesh-dnserror-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-dnserror-damengdb-mycwkx.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:37 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:36 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-mycwkx" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart damengdb-mycwkx --auto-approve --force=true --namespace ns-aihga ` OpsRequest damengdb-mycwkx-restart-7swlq created successfully, you can view the progress: kbcli cluster describe-ops damengdb-mycwkx-restart-7swlq -n ns-aihga check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-restart-7swlq ns-aihga Restart damengdb-mycwkx dmdb,dmmonitor Running 0/3 Sep 12,2025 13:40 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:40 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:41 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-restart-7swlq ns-aihga Restart damengdb-mycwkx dmdb,dmmonitor Succeed 3/3 Sep 12,2025 13:40 UTC+0800 check ops status done ops_status:damengdb-mycwkx-restart-7swlq ns-aihga Restart damengdb-mycwkx dmdb,dmmonitor Succeed 3/3 Sep 12,2025 13:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-restart-7swlq --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-restart-7swlq patched `kbcli cluster delete-ops --name damengdb-mycwkx-restart-7swlq --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-restart-7swlq deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover fullcpu check cluster status before cluster-failover-fullcpu check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-mycwkx" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-damengdb-mycwkx.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-damengdb-mycwkx created apply test-chaos-mesh-fullcpu-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-fullcpu-damengdb-mycwkx.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:40 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:41 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-mycwkx" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test switchover apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-mycwkx-switchover- namespace: ns-aihga spec: clusterName: damengdb-mycwkx type: Custom custom: components: - componentName: dmmonitor parameters: - name: candidate value: damengdb-mycwkx-dmdb-1 opsDefinitionName: switchover-for-dm `kubectl create -f test_create_damengdb-mycwkx_switchover.yaml` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-switchover-9bs9j created create test_create_damengdb-mycwkx_switchover.yaml Success `rm -rf test_create_damengdb-mycwkx_switchover.yaml` switchover pod:damengdb-mycwkx-dmdb-0 checking switchover... switchover pod:damengdb-mycwkx-dmdb-0 checking switchover... switchover pod:damengdb-mycwkx-dmdb-0 checking switchover... switchover pod:damengdb-mycwkx-dmdb-1 switchover success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-mycwkx" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 action: partition mode: all target: mode: all selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-damengdb-mycwkx.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-damengdb-mycwkx created apply test-chaos-mesh-networkpartition-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-networkpartition-damengdb-mycwkx.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:40 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:41 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-mycwkx" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-mycwkx" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-damengdb-mycwkx.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-damengdb-mycwkx created apply test-chaos-mesh-podfailure-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-podfailure-damengdb-mycwkx.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Abnormal Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:40 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:41 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-mycwkx" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster configure component_tmp: dmdb apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-mycwkx-reconfiguring- namespace: ns-aihga spec: type: Reconfiguring clusterName: damengdb-mycwkx force: true reconfigures: - componentName: dmdb parameters: - key: BUFFER value: '900' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-mycwkx.yaml` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-reconfiguring-qtzqf created create test_ops_cluster_damengdb-mycwkx.yaml Success `rm -rf test_ops_cluster_damengdb-mycwkx.yaml` check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-switchover-9bs9j ns-aihga Custom damengdb-mycwkx dmmonitor Succeed 1/1 Sep 12,2025 13:44 UTC+0800 damengdb-mycwkx-reconfiguring-qtzqf ns-aihga Reconfiguring damengdb-mycwkx dmdb,dmdb Running -/- Sep 12,2025 13:49 UTC+0800 check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Updating Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:50 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:49 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-mycwkx --status all --namespace ns-aihga ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-mycwkx-switchover-9bs9j ns-aihga Custom damengdb-mycwkx dmmonitor Succeed 1/1 Sep 12,2025 13:44 UTC+0800 damengdb-mycwkx-reconfiguring-qtzqf ns-aihga Reconfiguring damengdb-mycwkx dmdb,dmdb Succeed -/- Sep 12,2025 13:49 UTC+0800 check ops status done ops_status:damengdb-mycwkx-reconfiguring-qtzqf ns-aihga Reconfiguring damengdb-mycwkx dmdb,dmdb Succeed -/- Sep 12,2025 13:49 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations damengdb-mycwkx-reconfiguring-qtzqf --namespace ns-aihga ` opsrequest.operations.kubeblocks.io/damengdb-mycwkx-reconfiguring-qtzqf patched `kbcli cluster delete-ops --name damengdb-mycwkx-reconfiguring-qtzqf --force --auto-approve --namespace ns-aihga ` OpsRequest damengdb-mycwkx-reconfiguring-qtzqf deleted component_config:dmdb check config variables Defaulted container "dmdb" out of: dmdb, exporter, kbagent, config-manager, init-kbagent (init), kbagent-worker (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 900 configure:[BUFFER] result actual:[900] equal expected:[900] `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-mycwkx" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-damengdb-mycwkx.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-damengdb-mycwkx created apply test-chaos-mesh-networkdelay-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-networkdelay-damengdb-mycwkx.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:50 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:49 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-0;standby: damengdb-mycwkx-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-mycwkx" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-mycwkx" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-damengdb-mycwkx.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-damengdb-mycwkx created apply test-chaos-mesh-networkbandwidthover-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-damengdb-mycwkx.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:50 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:49 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-0 damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: No resources found in ns-aihga namespace. primary: damengdb-mycwkx-dmdb-1;standby: check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-mycwkx" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-mycwkx" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-mycwkx" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-damengdb-mycwkx namespace: ns-aihga spec: selector: namespaces: - ns-aihga labelSelectors: apps.kubeblocks.io/pod-name: damengdb-mycwkx-dmdb-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-damengdb-mycwkx.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-damengdb-mycwkx created apply test-chaos-mesh-networkduplicate-damengdb-mycwkx.yaml Success `rm -rf test-chaos-mesh-networkduplicate-damengdb-mycwkx.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:50 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:49 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-damengdb-mycwkx --namespace ns-aihga ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-mycwkx" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-mycwkx" not found check failover pod name failover pod name:damengdb-mycwkx-dmdb-1 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/75wHfEy7od@127.0.0.1:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-0 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update damengdb-mycwkx --termination-policy=WipeOut --namespace ns-aihga ` cluster.apps.kubeblocks.io/damengdb-mycwkx updated (no change) check cluster status `kbcli cluster list damengdb-mycwkx --show-labels --namespace ns-aihga ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-mycwkx ns-aihga damengdb WipeOut Running Sep 12,2025 13:05 UTC+0800 app.kubernetes.io/instance=damengdb-mycwkx,clusterdefinition.kubeblocks.io/name=damengdb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-mycwkx --namespace ns-aihga ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-mycwkx-dmdb-0 ns-aihga damengdb-mycwkx dmdb Running standby 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000002/10.224.0.6 Sep 12,2025 13:50 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmdb-1 ns-aihga damengdb-mycwkx dmdb Running primary 0 1100m / 1100m 2254857830400m / 2254857830400m data:25Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:49 UTC+0800 arch:20Gi log:20Gi damengdb-mycwkx-dmmonitor-0 ns-aihga damengdb-mycwkx dmmonitor Running 0 1 / 1 2Gi / 2Gi data:20Gi aks-cicdamdpool-13759654-vmss000000/10.224.0.5 Sep 12,2025 13:40 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-mycwkx-dmdb-1;standby: damengdb-mycwkx-dmdb-0 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-mycwkx` set secret: damengdb-mycwkx-dmdb-account-sysdba `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-mycwkx-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:75wHfEy7od;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-aihga namespace. check cluster connect `echo " ./disql sysdba/75wHfEy7od@damengdb-mycwkx-dmdb-rw.ns-aihga.svc.cluster.local:5236 " | kubectl exec -it damengdb-mycwkx-dmdb-1 --namespace ns-aihga -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done cluster list-logs `kbcli cluster list-logs damengdb-mycwkx --namespace ns-aihga ` No log files found. Error from server (NotFound): pods "damengdb-mycwkx-dmdb-1" not found cluster logs `kbcli cluster logs damengdb-mycwkx --tail 30 --namespace ns-aihga ` Defaulted container "dmdb" out of: dmdb, exporter, kbagent, config-manager, init-kbagent (init), kbagent-worker (init), install-config-manager-tool (init) 2025-09-12 05:55:58.818 [INFO] database P0000000599 T0000000000000000660 utsk_cmd_exec, utsk_process_mdw_chg_march success, sys_status:SUSPEND! 2025-09-12 05:55:58.818 [INFO] database P0000000599 T0000000000000000732 utsk_cmd_add, received sql exec cmd:1, dseq:1757656232, sql:ALTER DATABASE OPEN FORCE 2025-09-12 05:55:58.818 [INFO] database P0000000599 T0000000000000000732 utsk_cmd_add, cmd info: cmd=1, dseq=1757656232, name_in=, begin_lsn=-1! 2025-09-12 05:55:58.818 [INFO] database P0000000599 T0000000000000000732 utsk_dw_sql, before enqueue, len:0, tsks:0 2025-09-12 05:55:58.818 [INFO] database P0000000599 T0000000000000000732 utsk_dw_sql, after enqueue, len:1, tsks:1 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000732 utsk_dw_sql:ALTER DATABASE OPEN FORCE put to task sys, sys_status:SUSPEND 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 utsk_cmd_exec, cmd:1, sys_status:SUSPEND, dseq:1757656232 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 alter database open start... 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 altdb open llog_sys_open_or_close end! 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 altdb open rlog_flush_notify_ex start! 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 altdb open rlog_flush_notify_ex end! 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 alter database open success! 2025-09-12 05:55:58.819 [INFO] database P0000000599 T0000000000000000659 utsk_process_dw_sql_exec, prepare sql:ALTER DATABASE OPEN FORCE, code:0, dseq:1757656232 2025-09-12 05:55:59.819 [INFO] database P0000000599 T0000000000000000732 utsk_cmd_add, cmd info: cmd=217, dseq=1757656233, name_in=, begin_lsn=-1! 2025-09-12 05:55:59.819 [INFO] database P0000000599 T0000000000000000732 utsk_set_global_dw_stat, begin, msg_dseq:1757656233 2025-09-12 05:55:59.820 [INFO] database P0000000599 T0000000000000000732 set g_dw_stat from DW_RECOVERY to NONE success, g_dw_recover_stop is 0 2025-09-12 05:55:59.820 [INFO] database P0000000599 T0000000000000000732 Notify dest ep(DMDB_0) set dw_stat_flag from DW_RECOVERY to NONE success. 2025-09-12 05:55:59.820 [INFO] database P0000000599 T0000000000000000732 utsk_set_global_dw_stat, finished, msg_dseq:1757656233, set code:0 2025-09-12 05:56:03.746 [INFO] database P0000000599 T0000000000000002235 socket_err_should_retry errno:104 2025-09-12 05:56:08.200 [INFO] database P0000000599 T0000000000000002268 socket_err_should_retry errno:104 2025-09-12 05:56:13.594 [INFO] database P0000000599 T0000000000000000682 checkpoint requested by CKPT_INTERVAL, rlog free space[4294938624], used space[20480] 2025-09-12 05:56:13.594 [INFO] database P0000000599 T0000000000000000682 checkpoint generate by ckpt_interval 2025-09-12 05:56:13.594 [INFO] database P0000000599 T0000000000000000649 checkpoint begin, used_space[20480], free_space[4294938624]... 2025-09-12 05:56:13.595 [INFO] database P0000000599 T0000000000000000650 Local arch send type to DMDB_0 change from 1 to 0, clear recent arch send info! 2025-09-12 05:56:13.605 [INFO] database P0000000599 T0000000000000000649 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-09-12 05:56:13.605 [INFO] database P0000000599 T0000000000000000649 ckpt2_log_adjust: ckpt_lsn(109328), ckpt_fil(0), ckpt_off(39272448), cur_lsn(109630), l_next_seq(14355), g_next_seq(14282), cur_free(39297024), total_space(4294959104), used_space(24576), free_space(4294934528), n_ep(1) 2025-09-12 05:56:13.605 [INFO] database P0000000599 T0000000000000000649 checkpoint end, 0 pages flushed, used_space[24576], free_space[4294934528]. 2025-09-12 05:58:19.653 [INFO] database P0000000599 T0000000000000002833 socket_err_should_retry errno:104 2025-09-12 05:58:24.281 [INFO] database P0000000599 T0000000000000002859 socket_err_should_retry errno:104 2025-09-12 05:58:38.954 [INFO] database P0000000599 T0000000000000002931 socket_err_should_retry errno:104 delete cluster damengdb-mycwkx `kbcli cluster delete damengdb-mycwkx --auto-approve --namespace ns-aihga ` Cluster damengdb-mycwkx deleted secret "damengdb-mycwkx-license" deleted pod_info:damengdb-mycwkx-dmdb-0 4/4 Running 0 8m2s damengdb-mycwkx-dmdb-1 4/4 Running 0 8m46s damengdb-mycwkx-dmmonitor-0 1/1 Terminating 0 18m No resources found in ns-aihga namespace. delete cluster pod done No resources found in ns-aihga namespace. check cluster resource non-exist OK: pvc No resources found in ns-aihga namespace. delete cluster done No resources found in ns-aihga namespace. No resources found in ns-aihga namespace. No resources found in ns-aihga namespace. Damengdb Test Suite All Done! Test Engine: damengdb Test Type: 57 --------------------------------------Damengdb (Topology = realtime-replication Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=dmdb-repl-8.1.3-1.0.1;ComponentVersion=damengdb;ServiceVersion=8.1.3-26-20240821;]|[Description=Create a cluster with the specified component definition dmdb-repl-8.1.3-1.0.1 and component version damengdb and service version 8.1.3-26-20240821] [PASSED]|[Connect]|[ComponentName=dmdb]|[Description=Connect to the cluster] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=dmdb]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[VolumeExpansion]|[ComponentName=dmdb]|[Description=VolumeExpansion the cluster specify component dmdb] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=dmdb]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[HorizontalScaling Out]|[ComponentName=dmdb]|[Description=HorizontalScaling Out the cluster specify component dmdb] [PASSED]|[HorizontalScaling In]|[ComponentName=dmdb]|[Description=HorizontalScaling In the cluster specify component dmdb] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=dmdb]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=dmdb]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=dmdb]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[VerticalScaling]|[ComponentName=dmdb]|[Description=VerticalScaling the cluster specify component dmdb] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[SwitchOver]|[ComponentName=dmmonitor]|[Description=SwitchOver the cluster specify component dmmonitor] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=dmdb]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Reconfiguring]|[ComponentName=dmdb;BUFFER=900]|[Description=Reconfiguring the cluster specify component dmdb set BUFFER=900] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=dmdb]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=dmdb]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=dmdb]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]