source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-lsygw ` `kubectl create namespace ns-lsygw` namespace/ns-lsygw created create namespace ns-lsygw done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 32.1M 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 88.6M 0 --:--:-- --:--:-- --:--:-- 300M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:dmdb set component version set component version:damengdb set service versions:8.1.0,8.1.3-100-20240124,8.1.3-26-20240821,8.1.4-6-20241231 set service versions sorted:8.1.0,8.1.3-100-20240124,8.1.3-26-20240821,8.1.4-6-20241231 no cluster version found set damengdb component definition set damengdb component definition dmdb-repl-8.1.4 set replicas first:3,8.1.0|3,8.1.3-100-20240124|3,8.1.3-26-20240821 set replicas third:3,8.1.0 set replicas fourth:3,8.1.0 set minimum cmpv service version set minimum cmpv service version replicas:3,8.1.3-26-20240821 REPORT_COUNT:1 CLUSTER_TOPOLOGY:realtime-replication topology realtime-replication found in cluster definition damengdb LIMIT_CPU:1 LIMIT_MEMORY:2 storage size: 20 No resources found in ns-lsygw namespace. termination_policy:WipeOut create 3 replica WipeOut damengdb cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition1: dmdb-8.1 by component version:damengdb apiVersion: v1 kind: Secret metadata: name: damengdb-ghboje-license namespace: ns-lsygw type: Opaque --- apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: damengdb-ghboje namespace: ns-lsygw spec: clusterDefinitionRef: damengdb topology: realtime-replication terminationPolicy: WipeOut componentSpecs: - name: dmdb serviceVersion: 8.1.3-26-20240821 podUpdatePolicy: Recreate replicas: 3 env: - name: "OGUID" value: "1" - name: "UNICODE_FLAG" value: "1" - name: "LENGTH_IN_CHAR" value: "1" - name: "CASE_SENSITIVE" value: "0" - name: "BLANK_PAD_MODE" value: "0" - name: "ARCH_WAIT_APPLY" value: "0" - name: "HOST_NETWORK" value: "disabled" - name: "TIME_ZONE" value: "+08:00" - name: "ENABLE_AUDIT" value: "true" - name: "INSTANCE_NAME_PREFIX" value: "" - name: "PAGE_SIZE" value: "8" - name: "EXTENT_SIZE" value: "16" - name: "LOG_SIZE" value: "2048" - name: "SORT_MODE" value: "0" - name: "PK_CLUSTER_MODE" value: "0" - name: "ARCH_SPACE_LIMIT" value: "51200" - name: "IS_REMOTE_BACKUP" value: "false" - name: "REMOTE_TIMER_INTERVAL" value: "1" volumes: - name: license secret: secretName: damengdb-ghboje-license resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: arch spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: dmmonitor serviceVersion: 8.1.3-26-20240821 podUpdatePolicy: Recreate replicas: 1 env: - name: "OGUID" value: "1" resources: requests: cpu: 1000m memory: 2Gi limits: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi `kubectl apply -f test_create_damengdb-ghboje.yaml` secret/damengdb-ghboje-license created cluster.apps.kubeblocks.io/damengdb-ghboje created apply test_create_damengdb-ghboje.yaml Success `rm -rf test_create_damengdb-ghboje.yaml` check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Jun 19,2025 18:16 UTC+0800 clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:16 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:19 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:21 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba check pod damengdb-ghboje-dmdb-0 container_name dmdb exist password qeH4zy5c3z check pod damengdb-ghboje-dmdb-0 container_name exporter exist password qeH4zy5c3z check pod damengdb-ghboje-dmdb-0 container_name lorry exist password qeH4zy5c3z check pod damengdb-ghboje-dmdb-0 container_name config-manager exist password qeH4zy5c3z No container logs contain secret password. describe cluster `kbcli cluster describe damengdb-ghboje --namespace ns-lsygw ` Name: damengdb-ghboje Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-lsygw damengdb Running WipeOut Endpoints: COMPONENT MODE INTERNAL EXTERNAL dmdb ReadWrite damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME dmdb damengdb-ghboje-dmdb-0 primary Running us-west-2a ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:16 UTC+0800 dmdb damengdb-ghboje-dmdb-1 standby Running us-west-2a ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:19 UTC+0800 dmdb damengdb-ghboje-dmdb-2 standby Running us-west-2a ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:21 UTC+0800 dmmonitor damengdb-ghboje-dmmonitor-0 Running us-west-2a ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS dmdb false 1 / 1 2Gi / 2Gi data:20Gi kb-default-sc arch:20Gi kb-default-sc log:20Gi kb-default-sc dmmonitor false 1 / 1 2Gi / 2Gi data:20Gi kb-default-sc Images: COMPONENT TYPE IMAGE dmdb apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/dm:8.1.3-26 dmmonitor apecloud-registry.cn-zhangjiakou.cr.aliyuncs.com/apecloud/dm:8.1.3-26 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-lsygw damengdb-ghboje `kbcli cluster label damengdb-ghboje app.kubernetes.io/instance- --namespace ns-lsygw ` label "app.kubernetes.io/instance" not found. `kbcli cluster label damengdb-ghboje app.kubernetes.io/instance=damengdb-ghboje --namespace ns-lsygw ` `kbcli cluster label damengdb-ghboje --list --namespace ns-lsygw ` NAME NAMESPACE LABELS damengdb-ghboje ns-lsygw app.kubernetes.io/instance=damengdb-ghboje clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=damengdb-ghboje Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=damengdb-ghboje --namespace ns-lsygw ` `kbcli cluster label damengdb-ghboje --list --namespace ns-lsygw ` NAME NAMESPACE LABELS damengdb-ghboje ns-lsygw app.kubernetes.io/instance=damengdb-ghboje case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label damengdb-ghboje case.name=kbcli.test2 --overwrite --namespace ns-lsygw ` `kbcli cluster label damengdb-ghboje --list --namespace ns-lsygw ` NAME NAMESPACE LABELS damengdb-ghboje ns-lsygw app.kubernetes.io/instance=damengdb-ghboje case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label damengdb-ghboje case.name- --namespace ns-lsygw ` `kbcli cluster label damengdb-ghboje --list --namespace ns-lsygw ` NAME NAMESPACE LABELS damengdb-ghboje ns-lsygw app.kubernetes.io/instance=damengdb-ghboje clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect `echo 'cat /opt/dmdbms/data/DAMENG/dm.ini | grep MAX_SESSIONS' | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash ` Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file MAX_SESSIONS = 1500 #Maximum number of concurrent sessions connect cluster Success set max_connections to 1500 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-damengdb-ghboje" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-damengdb-ghboje --namespace ns-lsygw ` Error from server (NotFound): pods "test-db-client-executionloop-damengdb-ghboje" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-damengdb-ghboje" not found `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-damengdb-ghboje namespace: ns-lsygw spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local" - "--user" - "sysdba" - "--password" - "qeH4zy5c3z" - "--port" - "5236" - "--dbtype" - "damengdb" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-damengdb-ghboje.yaml` pod/test-db-client-executionloop-damengdb-ghboje created apply test-db-client-executionloop-damengdb-ghboje.yaml Success `rm -rf test-db-client-executionloop-damengdb-ghboje.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 1/1 Running 0 59s check pod test-db-client-executionloop-damengdb-ghboje status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-damengdb-ghboje 0/1 Completed 0 65s check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:16 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:19 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:21 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done --host damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local --user sysdba --password qeH4zy5c3z --port 5236 --dbtype damengdb --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:25:16.485 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: tableCount: 0 create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); Execution loop start:INSERT INTO executions_loop_table (value) VALUES ('executions_loop_test_1'); [ 1s ] executions total: 169 successful: 169 failed: 0 disconnect: 0 [ 2s ] executions total: 362 successful: 362 failed: 0 disconnect: 0 [ 3s ] executions total: 567 successful: 567 failed: 0 disconnect: 0 [ 4s ] executions total: 775 successful: 775 failed: 0 disconnect: 0 [ 5s ] executions total: 979 successful: 979 failed: 0 disconnect: 0 [ 6s ] executions total: 1189 successful: 1189 failed: 0 disconnect: 0 [ 7s ] executions total: 1361 successful: 1361 failed: 0 disconnect: 0 [ 8s ] executions total: 1588 successful: 1588 failed: 0 disconnect: 0 [ 9s ] executions total: 1817 successful: 1817 failed: 0 disconnect: 0 [ 10s ] executions total: 2017 successful: 2017 failed: 0 disconnect: 0 [ 11s ] executions total: 2226 successful: 2226 failed: 0 disconnect: 0 [ 12s ] executions total: 2447 successful: 2447 failed: 0 disconnect: 0 [ 13s ] executions total: 2653 successful: 2653 failed: 0 disconnect: 0 [ 14s ] executions total: 2856 successful: 2856 failed: 0 disconnect: 0 [ 15s ] executions total: 3059 successful: 3059 failed: 0 disconnect: 0 [ 16s ] executions total: 3274 successful: 3274 failed: 0 disconnect: 0 [ 17s ] executions total: 3480 successful: 3480 failed: 0 disconnect: 0 [ 18s ] executions total: 3667 successful: 3667 failed: 0 disconnect: 0 [ 19s ] executions total: 3876 successful: 3876 failed: 0 disconnect: 0 [ 20s ] executions total: 4080 successful: 4080 failed: 0 disconnect: 0 [ 21s ] executions total: 4277 successful: 4277 failed: 0 disconnect: 0 [ 22s ] executions total: 4482 successful: 4482 failed: 0 disconnect: 0 [ 23s ] executions total: 4685 successful: 4685 failed: 0 disconnect: 0 [ 24s ] executions total: 4898 successful: 4898 failed: 0 disconnect: 0 [ 25s ] executions total: 5129 successful: 5129 failed: 0 disconnect: 0 [ 26s ] executions total: 5319 successful: 5319 failed: 0 disconnect: 0 [ 27s ] executions total: 5550 successful: 5550 failed: 0 disconnect: 0 [ 28s ] executions total: 5769 successful: 5769 failed: 0 disconnect: 0 [ 29s ] executions total: 5985 successful: 5985 failed: 0 disconnect: 0 [ 30s ] executions total: 6206 successful: 6206 failed: 0 disconnect: 0 [ 31s ] executions total: 6413 successful: 6413 failed: 0 disconnect: 0 [ 32s ] executions total: 6630 successful: 6630 failed: 0 disconnect: 0 [ 33s ] executions total: 6856 successful: 6856 failed: 0 disconnect: 0 [ 34s ] executions total: 7072 successful: 7072 failed: 0 disconnect: 0 [ 35s ] executions total: 7286 successful: 7286 failed: 0 disconnect: 0 [ 36s ] executions total: 7513 successful: 7513 failed: 0 disconnect: 0 [ 37s ] executions total: 7720 successful: 7720 failed: 0 disconnect: 0 [ 38s ] executions total: 7944 successful: 7944 failed: 0 disconnect: 0 [ 39s ] executions total: 8161 successful: 8161 failed: 0 disconnect: 0 [ 40s ] executions total: 8380 successful: 8380 failed: 0 disconnect: 0 [ 41s ] executions total: 8599 successful: 8599 failed: 0 disconnect: 0 [ 42s ] executions total: 8814 successful: 8814 failed: 0 disconnect: 0 [ 43s ] executions total: 9059 successful: 9059 failed: 0 disconnect: 0 [ 44s ] executions total: 9281 successful: 9281 failed: 0 disconnect: 0 [ 45s ] executions total: 9475 successful: 9475 failed: 0 disconnect: 0 [ 46s ] executions total: 9659 successful: 9659 failed: 0 disconnect: 0 [ 47s ] executions total: 9874 successful: 9874 failed: 0 disconnect: 0 [ 48s ] executions total: 10094 successful: 10094 failed: 0 disconnect: 0 [ 49s ] executions total: 10316 successful: 10316 failed: 0 disconnect: 0 [ 50s ] executions total: 10523 successful: 10523 failed: 0 disconnect: 0 [ 51s ] executions total: 10712 successful: 10712 failed: 0 disconnect: 0 [ 52s ] executions total: 10926 successful: 10926 failed: 0 disconnect: 0 [ 53s ] executions total: 11145 successful: 11145 failed: 0 disconnect: 0 [ 54s ] executions total: 11354 successful: 11354 failed: 0 disconnect: 0 [ 55s ] executions total: 11564 successful: 11564 failed: 0 disconnect: 0 [ 56s ] executions total: 11780 successful: 11780 failed: 0 disconnect: 0 [ 57s ] executions total: 11981 successful: 11981 failed: 0 disconnect: 0 [ 58s ] executions total: 12201 successful: 12201 failed: 0 disconnect: 0 [ 59s ] executions total: 12411 successful: 12411 failed: 0 disconnect: 0 [ 60s ] executions total: 12589 successful: 12589 failed: 0 disconnect: 0 Test Result: Total Executions: 12589 Successful Executions: 12589 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: damengdb Host: damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local Port: 5236 Database: Table: User: sysdba Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 12589 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-damengdb-ghboje --namespace ns-lsygw ` pod/test-db-client-executionloop-damengdb-ghboje patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-damengdb-ghboje" force deleted test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-ghboje" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-damengdb-ghboje.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-damengdb-ghboje created apply test-chaos-mesh-networkdelay-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-networkdelay-damengdb-ghboje.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:16 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:19 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:21 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-ghboje" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-ghboje" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-damengdb-ghboje --namespace ns-lsygw ` Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-ghboje" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-damengdb-ghboje" not found `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-damengdb-ghboje namespace: ns-lsygw spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local" - "--user" - "sysdba" - "--password" - "qeH4zy5c3z" - "--port" - "5236" - "--database" - "sysdba" - "--dbtype" - "damengdb" - "--test" - "connectionstress" - "--connections" - "1500" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-damengdb-ghboje.yaml` pod/test-db-client-connectionstress-damengdb-ghboje created apply test-db-client-connectionstress-damengdb-ghboje.yaml Success `rm -rf test-db-client-connectionstress-damengdb-ghboje.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-damengdb-ghboje 1/1 Running 0 7s check pod test-db-client-connectionstress-damengdb-ghboje status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-damengdb-ghboje 0/1 Completed 0 12s check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:16 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:19 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:21 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done --host damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local --user sysdba --password qeH4zy5c3z --port 5236 --database sysdba --dbtype damengdb --test connectionstress --connections 1500 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:29:45.373 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Test Result: Created 1500 connections Connection Information: Database Type: damengdb Host: damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local Port: 5236 Database: sysdba Table: User: sysdba Org: Access Mode: mysql Test Type: connectionstress Connection Count: 1500 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-damengdb-ghboje --namespace ns-lsygw ` pod/test-db-client-connectionstress-damengdb-ghboje patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-damengdb-ghboje" force deleted check failover pod name failover pod name:damengdb-ghboje-dmdb-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=damengdb-ghboje,apps.kubeblocks.io/component-name=dmdb,apps.kubeblocks.io/vct-name=data --namespace ns-lsygw ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-ghboje namespace. `kbcli cluster volume-expand damengdb-ghboje --auto-approve --force=true --components dmdb --volume-claim-templates data --storage 22Gi --namespace ns-lsygw ` OpsRequest damengdb-ghboje-volumeexpansion-djkc8 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-volumeexpansion-djkc8 -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-volumeexpansion-djkc8 ns-lsygw VolumeExpansion damengdb-ghboje dmdb Running 0/3 Jun 19,2025 18:30 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1 / 1 2Gi / 2Gi data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:16 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:19 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1 / 1 2Gi / 2Gi data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:21 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-ghboje namespace. check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-volumeexpansion-djkc8 ns-lsygw VolumeExpansion damengdb-ghboje dmdb Succeed 3/3 Jun 19,2025 18:30 UTC+0800 check ops status done ops_status:damengdb-ghboje-volumeexpansion-djkc8 ns-lsygw VolumeExpansion damengdb-ghboje dmdb Succeed 3/3 Jun 19,2025 18:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-volumeexpansion-djkc8 --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-volumeexpansion-djkc8 patched `kbcli cluster delete-ops --name damengdb-ghboje-volumeexpansion-djkc8 --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-volumeexpansion-djkc8 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale damengdb-ghboje --auto-approve --force=true --components dmdb --cpu 1100m --memory 2.1Gi --namespace ns-lsygw ` OpsRequest damengdb-ghboje-verticalscaling-s9dlm created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-verticalscaling-s9dlm -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-verticalscaling-s9dlm ns-lsygw VerticalScaling damengdb-ghboje dmdb Running -/- Jun 19,2025 18:32 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:34 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:33 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:32 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-verticalscaling-s9dlm ns-lsygw VerticalScaling damengdb-ghboje dmdb Succeed 3/3 Jun 19,2025 18:32 UTC+0800 check ops status done ops_status:damengdb-ghboje-verticalscaling-s9dlm ns-lsygw VerticalScaling damengdb-ghboje dmdb Succeed 3/3 Jun 19,2025 18:32 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-verticalscaling-s9dlm --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-verticalscaling-s9dlm patched `kbcli cluster delete-ops --name damengdb-ghboje-verticalscaling-s9dlm --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-verticalscaling-s9dlm deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-ghboje namespace. `kbcli cluster hscale damengdb-ghboje --auto-approve --force=true --components dmdb --replicas 4 --namespace ns-lsygw ` OpsRequest damengdb-ghboje-horizontalscaling-5ccpm created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-horizontalscaling-5ccpm -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-horizontalscaling-5ccpm ns-lsygw HorizontalScaling damengdb-ghboje dmdb Running 0/1 Jun 19,2025 18:35 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:34 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:33 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:32 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-3 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:38 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-3 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-ghboje namespace. check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-horizontalscaling-5ccpm ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 18:35 UTC+0800 check ops status done ops_status:damengdb-ghboje-horizontalscaling-5ccpm ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 18:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-horizontalscaling-5ccpm --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-horizontalscaling-5ccpm patched `kbcli cluster delete-ops --name damengdb-ghboje-horizontalscaling-5ccpm --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-horizontalscaling-5ccpm deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in damengdb-ghboje namespace. `kbcli cluster hscale damengdb-ghboje --auto-approve --force=true --components dmdb --replicas 3 --namespace ns-lsygw ` OpsRequest damengdb-ghboje-horizontalscaling-5s6sd created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-horizontalscaling-5s6sd -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-horizontalscaling-5s6sd ns-lsygw HorizontalScaling damengdb-ghboje dmdb Running 0/1 Jun 19,2025 18:41 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:34 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:33 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:32 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done No resources found in damengdb-ghboje namespace. check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-horizontalscaling-5s6sd ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 18:41 UTC+0800 check ops status done ops_status:damengdb-ghboje-horizontalscaling-5s6sd ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 18:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-horizontalscaling-5s6sd --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-horizontalscaling-5s6sd patched `kbcli cluster delete-ops --name damengdb-ghboje-horizontalscaling-5s6sd --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-horizontalscaling-5s6sd deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-ghboje" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-2 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-damengdb-ghboje.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-damengdb-ghboje created apply test-chaos-mesh-dnserror-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-dnserror-damengdb-ghboje.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:34 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:33 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:32 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-ghboje" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-2 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-2 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-damengdb-ghboje.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-ghboje created apply test-chaos-mesh-oom-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-oom-damengdb-ghboje.yaml` check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:34 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:33 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:32 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-ghboje patched check failover pod name failover pod name:damengdb-ghboje-dmdb-2 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-2 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-damengdb-ghboje.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-ghboje created apply test-chaos-mesh-oom-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-oom-damengdb-ghboje.yaml` check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Abnormal Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:34 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:33 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:32 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-damengdb-ghboje" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-damengdb-ghboje patched check failover pod name:damengdb-ghboje-dmdb-0 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart damengdb-ghboje --auto-approve --force=true --namespace ns-lsygw ` OpsRequest damengdb-ghboje-restart-svjbh created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-restart-svjbh -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-restart-svjbh ns-lsygw Restart damengdb-ghboje dmdb,dmmonitor Running 0/4 Jun 19,2025 18:47 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:49 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:48 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:47 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-restart-svjbh ns-lsygw Restart damengdb-ghboje dmdb,dmmonitor Succeed 4/4 Jun 19,2025 18:47 UTC+0800 check ops status done ops_status:damengdb-ghboje-restart-svjbh ns-lsygw Restart damengdb-ghboje dmdb,dmmonitor Succeed 4/4 Jun 19,2025 18:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-restart-svjbh --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-restart-svjbh patched `kbcli cluster delete-ops --name damengdb-ghboje-restart-svjbh --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-restart-svjbh deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster configure component_tmp: dmdb check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure damengdb-ghboje --auto-approve --force=true --set BUFFER=900 --components dmdb --config-spec dmdb-config --config-file dm.ini --namespace ns-lsygw ` Will updated configure file meta: ConfigSpec: dmdb-config ConfigFile: dm.ini ComponentName: dmdb ClusterName: damengdb-ghboje OpsRequest damengdb-ghboje-reconfiguring-5v9k9 created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-reconfiguring-5v9k9 -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-reconfiguring-5v9k9 ns-lsygw Reconfiguring damengdb-ghboje dmdb Running -/- Jun 19,2025 18:50 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:51 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-reconfiguring-5v9k9 ns-lsygw Reconfiguring damengdb-ghboje dmdb Succeed 3/3 Jun 19,2025 18:50 UTC+0800 check ops status done ops_status:damengdb-ghboje-reconfiguring-5v9k9 ns-lsygw Reconfiguring damengdb-ghboje dmdb Succeed 3/3 Jun 19,2025 18:50 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-reconfiguring-5v9k9 --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-reconfiguring-5v9k9 patched `kbcli cluster delete-ops --name damengdb-ghboje-reconfiguring-5v9k9 --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-reconfiguring-5v9k9 deleted check config variables Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 900 configure:[BUFFER] result actual:[900] equal expected:[900] `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-ghboje" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-damengdb-ghboje.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-damengdb-ghboje created apply test-chaos-mesh-networkduplicate-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-networkduplicate-damengdb-ghboje.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:51 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-ghboje" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-ghboje" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-damengdb-ghboje.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-damengdb-ghboje created apply test-chaos-mesh-networklossover-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-networklossover-damengdb-ghboje.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:51 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-ghboje" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-1 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test switchover cluster promote apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-ghboje-switchover- namespace: ns-lsygw spec: clusterName: damengdb-ghboje custom: components: - componentName: dmmonitor parameters: - name: candidate value: damengdb-ghboje-dmdb-0 opsDefinitionName: switchover-for-dm type: Custom check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-ghboje.yaml` opsrequest.apps.kubeblocks.io/damengdb-ghboje-switchover-8s9cs created create test_ops_cluster_damengdb-ghboje.yaml Success `rm -rf test_ops_cluster_damengdb-ghboje.yaml` check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Running 0/1 Jun 19,2025 18:59 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:51 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Running 0/1 Jun 19,2025 18:59 UTC+0800 ops_status:damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Running 0/1 Jun 19,2025 18:59 UTC+0800 ops_status:damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Running 0/1 Jun 19,2025 18:59 UTC+0800 ops_status:damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Running 0/1 Jun 19,2025 18:59 UTC+0800 ops_status:damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Running 0/1 Jun 19,2025 18:59 UTC+0800 check ops status done ops_status:damengdb-ghboje-switchover-8s9cs ns-lsygw Custom damengdb-ghboje dmmonitor Succeed 1/1 Jun 19,2025 18:59 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-switchover-8s9cs --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-switchover-8s9cs patched `kbcli cluster delete-ops --name damengdb-ghboje-switchover-8s9cs --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-switchover-8s9cs deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success switchover pod:damengdb-ghboje-dmdb-0 switchover success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-ghboje" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-damengdb-ghboje.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-damengdb-ghboje created apply test-chaos-mesh-networkbandwidthover-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-damengdb-ghboje.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:51 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-ghboje" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-1 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Abnormal Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 18:51 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check failover pod name failover pod name:damengdb-ghboje-dmdb-0 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover delete pod:damengdb-ghboje-dmdb-0 `kubectl delete pod damengdb-ghboje-dmdb-0 --namespace ns-lsygw ` pod "damengdb-ghboje-dmdb-0" deleted check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:06 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:52 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check failover pod name failover pod name:damengdb-ghboje-dmdb-1 failover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod damengdb-ghboje-dmdb-1 --namespace ns-lsygw -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-1-97.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod damengdb-ghboje-dmdb-0 --namespace ns-lsygw -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-12-54.us-west-2.compute.internal success kubectl get pod damengdb-ghboje-dmdb-2 --namespace ns-lsygw -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-7-160.us-west-2.compute.internal success kubectl drain ip-172-31-1-97.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-1-97.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-zl5xt, kb-mqtcr/kb-addon-apecloud-otel-collector-n9l54, kube-system/aws-node-5gj5s, kube-system/ebs-csi-node-qw8bm, kube-system/kube-proxy-rb8xb evicting pod ns-lsygw/damengdb-ghboje-dmdb-1 pod/damengdb-ghboje-dmdb-1 evicted node/ip-172-31-1-97.us-west-2.compute.internal drained kubectl uncordon ip-172-31-1-97.us-west-2.compute.internal node/ip-172-31-1-97.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:06 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check failover pod name failover pod name:damengdb-ghboje-dmdb-0 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-ghboje" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-damengdb-ghboje.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-damengdb-ghboje created apply test-chaos-mesh-podfailure-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-podfailure-damengdb-ghboje.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:10 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-ghboje" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-2 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster update monitor KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster damengdb-ghboje --namespace ns-lsygw --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/damengdb-ghboje patched check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:10 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done create curl-jq pod damengdb-ghboje-kb-curl `kubectl run --namespace kb-mqtcr damengdb-ghboje-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/damengdb-ghboje-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE damengdb-ghboje-kb-curl 0/1 ContainerCreating 0 1s check pod damengdb-ghboje-kb-curl status done `kubectl wait --for=condition=Ready pod damengdb-ghboje-kb-curl --namespace kb-mqtcr` pod/damengdb-ghboje-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE damengdb-ghboje-kb-curl 1/1 Running 0 6s kubectl exec -it damengdb-ghboje-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it damengdb-ghboje-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 32937 0 --:--:-- --:--:-- --:--:-- 40666 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750331600.706,"24.800000000000004"]***]*** k8s_pod_cpu_limit: 1750331600.706 monitor cluster Success `kubectl delete pod damengdb-ghboje-kb-curl --namespace kb-mqtcr --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "damengdb-ghboje-kb-curl" force deleted check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:10 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-ghboje" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-2 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-damengdb-ghboje.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-damengdb-ghboje created apply test-chaos-mesh-timeoffset-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-timeoffset-damengdb-ghboje.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:10 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-ghboje" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-2 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-ghboje" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-2 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-damengdb-ghboje.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-damengdb-ghboje created apply test-chaos-mesh-dnsrandom-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-dnsrandom-damengdb-ghboje.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:10 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-ghboje" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-2 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale offline instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-ghboje-hscaleoffinstance- labels: app.kubernetes.io/instance: damengdb-ghboje app.kubernetes.io/managed-by: kubeblocks namespace: ns-lsygw spec: type: HorizontalScaling clusterName: damengdb-ghboje force: true horizontalScaling: - componentName: dmdb scaleIn: onlineInstancesToOffline: - damengdb-ghboje-dmdb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-ghboje.yaml` opsrequest.apps.kubeblocks.io/damengdb-ghboje-hscaleoffinstance-5f4g7 created create test_ops_cluster_damengdb-ghboje.yaml Success `rm -rf test_ops_cluster_damengdb-ghboje.yaml` check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-hscaleoffinstance-5f4g7 ns-lsygw HorizontalScaling damengdb-ghboje dmdb Running 0/1 Jun 19,2025 19:19 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-hscaleoffinstance-5f4g7 ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 19:19 UTC+0800 check ops status done ops_status:damengdb-ghboje-hscaleoffinstance-5f4g7 ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 19:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-hscaleoffinstance-5f4g7 --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-hscaleoffinstance-5f4g7 patched `kbcli cluster delete-ops --name damengdb-ghboje-hscaleoffinstance-5f4g7 --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-hscaleoffinstance-5f4g7 deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster hscale online instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: damengdb-ghboje-hscaleoninstance- labels: app.kubernetes.io/instance: damengdb-ghboje app.kubernetes.io/managed-by: kubeblocks namespace: ns-lsygw spec: type: HorizontalScaling clusterName: damengdb-ghboje force: true horizontalScaling: - componentName: dmdb scaleOut: offlineInstancesToOnline: - damengdb-ghboje-dmdb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_damengdb-ghboje.yaml` opsrequest.apps.kubeblocks.io/damengdb-ghboje-hscaleoninstance-m99wl created create test_ops_cluster_damengdb-ghboje.yaml Success `rm -rf test_ops_cluster_damengdb-ghboje.yaml` check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-hscaleoninstance-m99wl ns-lsygw HorizontalScaling damengdb-ghboje dmdb Creating -/- Jun 19,2025 19:20 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:22 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:07 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 18:50 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 18:47 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-2;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-1 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-hscaleoninstance-m99wl ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 19:20 UTC+0800 check ops status done ops_status:damengdb-ghboje-hscaleoninstance-m99wl ns-lsygw HorizontalScaling damengdb-ghboje dmdb Succeed 1/1 Jun 19,2025 19:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-hscaleoninstance-m99wl --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-hscaleoninstance-m99wl patched `kbcli cluster delete-ops --name damengdb-ghboje-hscaleoninstance-m99wl --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-hscaleoninstance-m99wl deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-2 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop damengdb-ghboje --auto-approve --force=true --namespace ns-lsygw ` OpsRequest damengdb-ghboje-stop-rqxbb created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-stop-rqxbb -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-stop-rqxbb ns-lsygw Stop damengdb-ghboje Creating -/- Jun 19,2025 19:23 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Stopping Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-stop-rqxbb ns-lsygw Stop damengdb-ghboje dmdb,dmmonitor Succeed 4/4 Jun 19,2025 19:23 UTC+0800 check ops status done ops_status:damengdb-ghboje-stop-rqxbb ns-lsygw Stop damengdb-ghboje dmdb,dmmonitor Succeed 4/4 Jun 19,2025 19:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-stop-rqxbb --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-stop-rqxbb patched `kbcli cluster delete-ops --name damengdb-ghboje-stop-rqxbb --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-stop-rqxbb deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start damengdb-ghboje --force=true --namespace ns-lsygw ` OpsRequest damengdb-ghboje-start-tlfct created successfully, you can view the progress: kbcli cluster describe-ops damengdb-ghboje-start-tlfct -n ns-lsygw check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-start-tlfct ns-lsygw Start damengdb-ghboje dmdb,dmmonitor Running 0/4 Jun 19,2025 19:24 UTC+0800 check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:24 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done check ops status `kbcli cluster list-ops damengdb-ghboje --status all --namespace ns-lsygw ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME damengdb-ghboje-start-tlfct ns-lsygw Start damengdb-ghboje dmdb,dmmonitor Succeed 4/4 Jun 19,2025 19:24 UTC+0800 check ops status done ops_status:damengdb-ghboje-start-tlfct ns-lsygw Start damengdb-ghboje dmdb,dmmonitor Succeed 4/4 Jun 19,2025 19:24 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests damengdb-ghboje-start-tlfct --namespace ns-lsygw ` opsrequest.apps.kubeblocks.io/damengdb-ghboje-start-tlfct patched `kbcli cluster delete-ops --name damengdb-ghboje-start-tlfct --force --auto-approve --namespace ns-lsygw ` OpsRequest damengdb-ghboje-start-tlfct deleted `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-damengdb-ghboje.yaml` Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-ghboje" not found stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-damengdb-ghboje created apply test-chaos-mesh-fullcpu-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-fullcpu-damengdb-ghboje.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:24 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-0;standby: damengdb-ghboje-dmdb-1 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-ghboje" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-0 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-ghboje" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-damengdb-ghboje.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-damengdb-ghboje created apply test-chaos-mesh-networkcorruptover-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-damengdb-ghboje.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:24 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-ghboje" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-1 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-ghboje" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-ghboje" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-damengdb-ghboje namespace: ns-lsygw spec: selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-1 action: partition mode: all target: mode: all selector: namespaces: - ns-lsygw labelSelectors: apps.kubeblocks.io/pod-name: damengdb-ghboje-dmdb-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-damengdb-ghboje.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-damengdb-ghboje created apply test-chaos-mesh-networkpartition-damengdb-ghboje.yaml Success `rm -rf test-chaos-mesh-networkpartition-damengdb-ghboje.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:24 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-damengdb-ghboje --namespace ns-lsygw ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-ghboje" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-damengdb-ghboje" not found check failover pod name failover pod name:damengdb-ghboje-dmdb-1 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check readonly db_client batch data count `echo " echo \"select count(*) from executions_loop_table;\" | ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-ro.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-0 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update damengdb-ghboje --termination-policy=WipeOut --namespace ns-lsygw ` cluster.apps.kubeblocks.io/damengdb-ghboje updated (no change) check cluster status `kbcli cluster list damengdb-ghboje --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje ns-lsygw damengdb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=damengdb-ghboje,clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-dmdb-0 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-12-54.us-west-2.compute.internal/172.31.12.54 Jun 19,2025 19:24 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-1 ns-lsygw damengdb-ghboje dmdb Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmdb-2 ns-lsygw damengdb-ghboje dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-dmmonitor-0 ns-lsygw damengdb-ghboje dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-7-160.us-west-2.compute.internal/172.31.7.160 Jun 19,2025 19:24 UTC+0800 check pod status done check cluster role check cluster role done primary: damengdb-ghboje-dmdb-1;standby: damengdb-ghboje-dmdb-0 damengdb-ghboje-dmdb-2 `kubectl get secrets -l app.kubernetes.io/instance=damengdb-ghboje` set secret: damengdb-ghboje-dmdb-account-sysdba `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.username***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.password***"` `kubectl get secrets damengdb-ghboje-dmdb-account-sysdba -o jsonpath="***.data.port***"` DB_USERNAME:sysdba;DB_PASSWORD:qeH4zy5c3z;DB_PORT:5236;DB_DATABASE:sysdba No resources found in ns-lsygw namespace. check cluster connect `echo " ./disql sysdba/qeH4zy5c3z@damengdb-ghboje-dmdb-rw.ns-lsygw.svc.cluster.local:5236 " | kubectl exec -it damengdb-ghboje-dmdb-1 --namespace ns-lsygw -- bash -c "cd /opt/dmdbms/bin && exec bash" ` check cluster connect done cluster full backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup damengdb-ghboje --method full --namespace ns-lsygw ` Backup backup-ns-lsygw-damengdb-ghboje-20250619193622 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-lsygw-damengdb-ghboje-20250619193622 -n ns-lsygw check backup status `kbcli cluster list-backups damengdb-ghboje --namespace ns-lsygw ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-lsygw-damengdb-ghboje-20250619193622 ns-lsygw damengdb-ghboje full Running Jun 19,2025 19:36 UTC+0800 backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running backup_status:damengdb-ghboje-full-Running check backup status done backup_status:backup-ns-lsygw-damengdb-ghboje-20250619193622 ns-lsygw damengdb-ghboje full Completed 1055971 110s Jun 19,2025 19:36 UTC+0800 Jun 19,2025 19:38 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "damengdb-ghboje-backup" not found `kbcli cluster describe-backup backup-ns-lsygw-damengdb-ghboje-20250619193622 --namespace ns-lsygw ` Name: backup-ns-lsygw-damengdb-ghboje-20250619193622 Cluster: damengdb-ghboje Namespace: ns-lsygw Spec: Method: full Policy Name: damengdb-ghboje-dmdb-backup-policy Status: Phase: Completed Total Size: 1055971 ActionSet Name: dmdb-full-backup Repository: backuprepo-kbcli-test Duration: 110s Start Time: Jun 19,2025 19:36 UTC+0800 Completion Time: Jun 19,2025 19:38 UTC+0800 Path: /ns-lsygw/damengdb-ghboje-945ff7e3-e537-481d-ae23-00a5ce88284d/dmdb/backup-ns-lsygw-damengdb-ghboje-20250619193622 Time Range Start: Jun 19,2025 19:30 UTC+0800 Time Range End: Jun 19,2025 19:38 UTC+0800 Warning Events: `kbcli cluster restore damengdb-ghboje-backup --backup backup-ns-lsygw-damengdb-ghboje-20250619193622 --namespace ns-lsygw ` Cluster damengdb-ghboje-backup created check cluster status `kbcli cluster list damengdb-ghboje-backup --show-labels --namespace ns-lsygw ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS damengdb-ghboje-backup ns-lsygw damengdb WipeOut Jun 19,2025 19:38 UTC+0800 clusterdefinition.kubeblocks.io/name=damengdb,clusterversion.kubeblocks.io/name= cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster damengdb-ghboje-backup yaml-------------------------------------- `kubectl get cluster damengdb-ghboje-backup -o yaml --namespace ns-lsygw ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"damengdb-ghboje-backup","type":"Restore"***]' kubeblocks.io/reconcile: "2025-06-19T11:31:30.239503573Z" kubeblocks.io/restore-from-backup: '***"dmdb":***"doReadyRestoreAfterClusterRunning":"false","name":"backup-ns-lsygw-damengdb-ghboje-20250619193622","namespace":"ns-lsygw","volumeRestorePolicy":"Parallel"***' creationTimestamp: "2025-06-19T11:38:21Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 1 labels: clusterdefinition.kubeblocks.io/name: damengdb clusterversion.kubeblocks.io/name: "" name: damengdb-ghboje-backup namespace: ns-lsygw resourceVersion: "110432" uid: dcb8a7f9-76ef-46f0-bb28-b567bdc7013c spec: clusterDefinitionRef: damengdb componentSpecs: - componentDef: dmdb-8.1 disableExporter: false env: - name: OGUID value: "1" - name: UNICODE_FLAG value: "1" - name: LENGTH_IN_CHAR value: "1" - name: CASE_SENSITIVE value: "0" - name: BLANK_PAD_MODE value: "0" - name: ARCH_WAIT_APPLY value: "0" - name: HOST_NETWORK value: disabled - name: TIME_ZONE value: "+08:00" - name: ENABLE_AUDIT value: "true" - name: INSTANCE_NAME_PREFIX - name: PAGE_SIZE value: "8" - name: EXTENT_SIZE value: "16" - name: LOG_SIZE value: "2048" - name: SORT_MODE value: "0" - name: PK_CLUSTER_MODE value: "0" - name: ARCH_SPACE_LIMIT value: "51200" - name: IS_REMOTE_BACKUP value: "false" - name: REMOTE_TIMER_INTERVAL value: "1" name: dmdb podUpdatePolicy: Recreate replicas: 3 resources: limits: cpu: 1100m memory: 2254857830400m requests: cpu: 1100m memory: 2254857830400m serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 22Gi - name: arch spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: log spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi volumes: - name: license secret: secretName: damengdb-ghboje-license - componentDef: dmmonitor-8.1 env: - name: OGUID value: "1" name: dmmonitor podUpdatePolicy: Recreate replicas: 1 resources: limits: cpu: "1" memory: 2Gi requests: cpu: "1" memory: 2Gi serviceVersion: 8.1.3-26-20240821 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: WipeOut topology: realtime-replication status: clusterDefGeneration: 2 components: dmdb: phase: Creating podsReady: false dmmonitor: phase: Running podsReady: true podsReadyTime: "2025-06-19T11:38:35Z" conditions: - lastTransitionTime: "2025-06-19T11:38:21Z" message: 'The operator has started the provisioning of Cluster: damengdb-ghboje-backup' observedGeneration: 1 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T11:38:21Z" message: Successfully applied for resources observedGeneration: 1 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:38:21Z" message: 'pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:38:21Z" message: 'pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 1 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster damengdb-ghboje-backup-------------------------------------- `kubectl describe cluster damengdb-ghboje-backup --namespace ns-lsygw ` Name: damengdb-ghboje-backup Namespace: ns-lsygw Labels: clusterdefinition.kubeblocks.io/name=damengdb clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"damengdb-ghboje-backup","type":"Restore"***] kubeblocks.io/reconcile: 2025-06-19T11:31:30.239503573Z kubeblocks.io/restore-from-backup: ***"dmdb":***"doReadyRestoreAfterClusterRunning":"false","name":"backup-ns-lsygw-damengdb-ghboje-20250619193622","namespace":"ns-lsygw","volum... API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T11:38:21Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 1 Resource Version: 110432 UID: dcb8a7f9-76ef-46f0-bb28-b567bdc7013c Spec: Cluster Definition Ref: damengdb Component Specs: Component Def: dmdb-8.1 Disable Exporter: false Env: Name: OGUID Value: 1 Name: UNICODE_FLAG Value: 1 Name: LENGTH_IN_CHAR Value: 1 Name: CASE_SENSITIVE Value: 0 Name: BLANK_PAD_MODE Value: 0 Name: ARCH_WAIT_APPLY Value: 0 Name: HOST_NETWORK Value: disabled Name: TIME_ZONE Value: +08:00 Name: ENABLE_AUDIT Value: true Name: INSTANCE_NAME_PREFIX Name: PAGE_SIZE Value: 8 Name: EXTENT_SIZE Value: 16 Name: LOG_SIZE Value: 2048 Name: SORT_MODE Value: 0 Name: PK_CLUSTER_MODE Value: 0 Name: ARCH_SPACE_LIMIT Value: 51200 Name: IS_REMOTE_BACKUP Value: false Name: REMOTE_TIMER_INTERVAL Value: 1 Name: dmdb Pod Update Policy: Recreate Replicas: 3 Resources: Limits: Cpu: 1100m Memory: 2254857830400m Requests: Cpu: 1100m Memory: 2254857830400m Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 22Gi Name: arch Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Volumes: Name: license Secret: Secret Name: damengdb-ghboje-license Component Def: dmmonitor-8.1 Env: Name: OGUID Value: 1 Name: dmmonitor Pod Update Policy: Recreate Replicas: 1 Resources: Limits: Cpu: 1 Memory: 2Gi Requests: Cpu: 1 Memory: 2Gi Service Version: 8.1.3-26-20240821 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: WipeOut Topology: realtime-replication Status: Cluster Def Generation: 2 Components: Dmdb: Phase: Creating Pods Ready: false Dmmonitor: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T11:38:35Z Conditions: Last Transition Time: 2025-06-19T11:38:21Z Message: The operator has started the provisioning of Cluster: damengdb-ghboje-backup Observed Generation: 1 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T11:38:21Z Message: Successfully applied for resources Observed Generation: 1 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:38:21Z Message: pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:38:21Z Message: pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 1 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 8m22s cluster-controller The operator has started the provisioning of Cluster: damengdb-ghboje-backup Normal ApplyResourcesSucceed 8m22s cluster-controller Successfully applied for resources Normal NeedWaiting 8m16s (x4 over 8m22s) component-controller waiting for restore "damengdb-ghboje-backup-dmdb-dcb8a7f9-preparedata" successfully Normal ComponentPhaseTransition 8m8s cluster-controller component is Running Warning ReplicasNotReady 8m8s cluster-controller pods are not ready in Components: [dmdb], refer to related component message in Cluster.status.components Warning ComponentsNotReady 8m8s cluster-controller pods are unavailable in Components: [dmdb], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 6m51s (x2 over 8m16s) cluster-controller component is Creating ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances damengdb-ghboje-backup --namespace ns-lsygw ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME damengdb-ghboje-backup-dmdb-0 ns-lsygw damengdb-ghboje-backup dmdb Running us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-3-13.us-west-2.compute.internal/172.31.3.13 Jun 19,2025 19:39 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-backup-dmdb-1 ns-lsygw damengdb-ghboje-backup dmdb Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 19:42 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-backup-dmdb-2 ns-lsygw damengdb-ghboje-backup dmdb Running us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m data:22Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:44 UTC+0800 arch:20Gi log:20Gi damengdb-ghboje-backup-dmmonitor-0 ns-lsygw damengdb-ghboje-backup dmmonitor Running us-west-2a 1 / 1 2Gi / 2Gi data:20Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 19:38 UTC+0800 check pod status done `kbcli cluster describe-backup backup-ns-lsygw-damengdb-ghboje-20250619193622 --namespace ns-lsygw ` Name: backup-ns-lsygw-damengdb-ghboje-20250619193622 Cluster: damengdb-ghboje Namespace: ns-lsygw Spec: Method: full Policy Name: damengdb-ghboje-dmdb-backup-policy Status: Phase: Completed Total Size: 1055971 ActionSet Name: dmdb-full-backup Repository: backuprepo-kbcli-test Duration: 110s Start Time: Jun 19,2025 19:36 UTC+0800 Completion Time: Jun 19,2025 19:38 UTC+0800 Path: /ns-lsygw/damengdb-ghboje-945ff7e3-e537-481d-ae23-00a5ce88284d/dmdb/backup-ns-lsygw-damengdb-ghboje-20250619193622 Time Range Start: Jun 19,2025 19:30 UTC+0800 Time Range End: Jun 19,2025 19:38 UTC+0800 Warning Events: delete cluster damengdb-ghboje-backup `kbcli cluster delete damengdb-ghboje-backup --auto-approve --namespace ns-lsygw ` Cluster damengdb-ghboje-backup deleted Error from server (NotFound): secrets "damengdb-ghboje-backup-license" not found Error from server (NotFound): secrets "damengdb-ghboje-backup-license" not found Error from server (NotFound): secrets "damengdb-ghboje-backup-license" not found pod_info:damengdb-ghboje-backup-dmdb-0 3/4 Terminating 0 7m4s damengdb-ghboje-backup-dmdb-1 3/4 Terminating 0 4m47s damengdb-ghboje-backup-dmdb-2 4/4 Terminating 0 2m9s No resources found in ns-lsygw namespace. delete cluster pod done No resources found in ns-lsygw namespace. check cluster resource non-exist OK: pvc No resources found in ns-lsygw namespace. delete cluster done No resources found in ns-lsygw namespace. No resources found in ns-lsygw namespace. No resources found in ns-lsygw namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-lsygw-damengdb-ghboje-20250619193622 --namespace ns-lsygw ` backup.dataprotection.kubeblocks.io/backup-ns-lsygw-damengdb-ghboje-20250619193622 patched `kbcli cluster delete-backup damengdb-ghboje --name backup-ns-lsygw-damengdb-ghboje-20250619193622 --force --auto-approve --namespace ns-lsygw ` Backup backup-ns-lsygw-damengdb-ghboje-20250619193622 deleted No opsrequests found in ns-lsygw namespace. cluster list-logs `kbcli cluster list-logs damengdb-ghboje --namespace ns-lsygw ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update damengdb-ghboje --enable-all-logs=true --namespace ns-lsygw Error from server (NotFound): pods "damengdb-ghboje-dmdb-1" not found cluster logs `kbcli cluster logs damengdb-ghboje --tail 30 --namespace ns-lsygw ` Defaulted container "dmdb" out of: dmdb, exporter, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) 2025-06-19 11:37:57.487 [INFO] database P0000000609 T0000000000000003939 buf4_wait_for_flush end clsn[132986] 2025-06-19 11:37:57.487 [INFO] database P0000000609 T0000000000000003939 hpc_begin_apply_info_get_online p_db_magic(0), n_apply_ep(0), pkg_seq_arr([17542, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), lsn_arr([132979, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 2025-06-19 11:37:57.487 [INFO] database P0000000609 T0000000000000003939 rfil[/opt/dmdbms/data/DAMENG/DAMENG01.log], first local_rpkg_seqno = 17630, global_rpkg_seqno = 17543, first_startup = 0 2025-06-19 11:37:57.505 [INFO] database P0000000609 T0000000000000003939 hpc_begin_apply_info_get_online p_db_magic(0), n_apply_ep(1), pkg_seq_arr([17543, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]), lsn_arr([132986, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]) 2025-06-19 11:38:03.018 [INFO] database P0000000609 T0000000000000003939 bak2_dbf_backup_exec finished, code:0, sess[0x7f4158011978], memobj[0x7f4158011980] 2025-06-19 11:38:03.045 [INFO] database P0000000609 T0000000000000003939 socket_err_should_retry errno:104 2025-06-19 11:38:04.228 [INFO] database P0000000609 T0000000000000000657 rafil_close_with_trunc file[/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x53B0D370_EP0_2025-06-19_11-30-17.log] to 57344 len, g_next_seq:17554, ret = 1! 2025-06-19 11:38:04.228 [INFO] database P0000000609 T0000000000000000657 rfil_close_low set arch rfil[/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x53B0D370_EP0_2025-06-19_11-30-17.log]'s sta to inactive, l_next_seq = 17554, g_next_seq = 17554, clsn = 133169, handle = 12, free=57344, len=57344 2025-06-19 11:40:57.827 [INFO] database P0000000609 T0000000000000000696 checkpoint requested by CKPT_INTERVAL, rlog free space[4294905856], used space[53248] 2025-06-19 11:40:57.827 [INFO] database P0000000609 T0000000000000000696 checkpoint generate by ckpt_interval 2025-06-19 11:40:57.827 [INFO] database P0000000609 T0000000000000000663 checkpoint begin, used_space[53248], free_space[4294905856]... 2025-06-19 11:40:57.830 [INFO] database P0000000609 T0000000000000000663 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-06-19 11:40:57.830 [INFO] database P0000000609 T0000000000000000663 ckpt2_log_adjust: ckpt_lsn(133141), ckpt_fil(0), ckpt_off(52731904), cur_lsn(133169), l_next_seq(17642), g_next_seq(17555), cur_free(52768768), total_space(4294959104), used_space(36864), free_space(4294922240), n_ep(1) 2025-06-19 11:40:57.830 [INFO] database P0000000609 T0000000000000000663 checkpoint end, 0 pages flushed, used_space[36864], free_space[4294922240]. 2025-06-19 11:43:57.904 [INFO] database P0000000609 T0000000000000000696 checkpoint requested by CKPT_INTERVAL, rlog free space[4294922240], used space[36864] 2025-06-19 11:43:57.904 [INFO] database P0000000609 T0000000000000000696 checkpoint generate by ckpt_interval 2025-06-19 11:43:57.904 [INFO] database P0000000609 T0000000000000000663 checkpoint begin, used_space[36864], free_space[4294922240]... 2025-06-19 11:43:57.908 [INFO] database P0000000609 T0000000000000000657 rafil_file_create [/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x53B0D370_EP0_2025-06-19_11-43-57.log] start 2025-06-19 11:43:57.909 [INFO] database P0000000609 T0000000000000000657 rfil_close_low set arch rfil[/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x53B0D370_EP0_2025-06-19_11-43-57.log]'s sta to inactive, l_next_seq = 2457, g_next_seq = 2457, clsn = 0, handle = 12, free=4096, len=134217728 2025-06-19 11:43:57.912 [INFO] database P0000000609 T0000000000000000657 rafil_file_create [/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x53B0D370_EP0_2025-06-19_11-43-57.log] success, handle = 12 2025-06-19 11:43:57.912 [INFO] database P0000000609 T0000000000000000657 rafil_file_init, rac_seqno:0, create new file and set last_arch_fil:/opt/dmdbms/arch/ARCHIVE_LOCAL1_0x53B0D370_EP0_2025-06-19_11-43-57.log! 2025-06-19 11:43:57.916 [INFO] database P0000000609 T0000000000000000663 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-06-19 11:43:57.916 [INFO] database P0000000609 T0000000000000000663 ckpt2_log_adjust: ckpt_lsn(133142), ckpt_fil(0), ckpt_off(52731904), cur_lsn(133169), l_next_seq(17643), g_next_seq(17556), cur_free(52772864), total_space(4294959104), used_space(40960), free_space(4294918144), n_ep(1) 2025-06-19 11:43:57.916 [INFO] database P0000000609 T0000000000000000663 checkpoint end, 0 pages flushed, used_space[40960], free_space[4294918144]. 2025-06-19 11:46:57.991 [INFO] database P0000000609 T0000000000000000696 checkpoint requested by CKPT_INTERVAL, rlog free space[4294918144], used space[40960] 2025-06-19 11:46:57.992 [INFO] database P0000000609 T0000000000000000696 checkpoint generate by ckpt_interval 2025-06-19 11:46:57.992 [INFO] database P0000000609 T0000000000000000663 checkpoint begin, used_space[40960], free_space[4294918144]... 2025-06-19 11:46:57.999 [INFO] database P0000000609 T0000000000000000663 ckpt2_log_adjust: full_status: 160, ptx_reserved: 0 2025-06-19 11:46:57.999 [INFO] database P0000000609 T0000000000000000663 ckpt2_log_adjust: ckpt_lsn(133143), ckpt_fil(0), ckpt_off(52731904), cur_lsn(133169), l_next_seq(17644), g_next_seq(17557), cur_free(52776960), total_space(4294959104), used_space(45056), free_space(4294914048), n_ep(1) 2025-06-19 11:46:57.999 [INFO] database P0000000609 T0000000000000000663 checkpoint end, 0 pages flushed, used_space[45056], free_space[4294914048]. delete cluster damengdb-ghboje `kbcli cluster delete damengdb-ghboje --auto-approve --namespace ns-lsygw ` Cluster damengdb-ghboje deleted secret "damengdb-ghboje-license" deleted pod_info:damengdb-ghboje-dmdb-0 4/4 Terminating 0 22m damengdb-ghboje-dmdb-1 4/4 Terminating 0 22m damengdb-ghboje-dmdb-2 4/4 Terminating 0 21m damengdb-ghboje-dmmonitor-0 1/1 Terminating 0 22m No resources found in ns-lsygw namespace. delete cluster pod done No resources found in ns-lsygw namespace. check cluster resource non-exist OK: pvc No resources found in ns-lsygw namespace. delete cluster done No resources found in ns-lsygw namespace. No resources found in ns-lsygw namespace. No resources found in ns-lsygw namespace. Damengdb Test Suite All Done! --------------------------------------Damengdb Ent (Topology = realtime-replication Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=realtime-replication;ComponentVersion=damengdb;ServiceVersion=8.1.3-26-20240821;]|[Description=Create a cluster with the specified topology realtime-replication and component version damengdb and service version 8.1.3-26-20240821] [PASSED]|[Connect]|[ComponentName=dmdb]|[Description=Connect to the cluster] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=dmdb]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=dmdb]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[VolumeExpansion]|[ComponentName=dmdb]|[Description=VolumeExpansion the cluster specify component dmdb] [PASSED]|[VerticalScaling]|[ComponentName=dmdb]|[Description=VerticalScaling the cluster specify component dmdb] [PASSED]|[HorizontalScaling Out]|[ComponentName=dmdb]|[Description=HorizontalScaling Out the cluster specify component dmdb] [PASSED]|[HorizontalScaling In]|[ComponentName=dmdb]|[Description=HorizontalScaling In the cluster specify component dmdb] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Reconfiguring]|[ComponentName=dmdb;BUFFER=900]|[Description=Reconfiguring the cluster specify component dmdb set BUFFER=900] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=dmdb]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=dmdb]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[SwitchOver]|[ComponentName=dmdb]|[Description=SwitchOver the cluster specify component dmdb] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=dmdb]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=dmdb]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=dmdb]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=dmdb]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=dmdb]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[HscaleOfflineInstances]|[ComponentName=dmdb]|[Description=Hscale the cluster instances offline specify component dmdb] [PASSED]|[HscaleOnlineInstances]|[ComponentName=dmdb]|[Description=Hscale the cluster instances online specify component dmdb] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=dmdb]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=dmdb]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=dmdb]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=full]|[Description=The cluster full Backup] [FAILED]|[Restore]|[BackupMethod=full]|[Description=The cluster full Restore] [PASSED]|[Delete Restore Cluster]|[BackupMethod=full]|[Description=Delete the full restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]