source commons files source engines files source kubeblocks files source kubedb files CLUSTER_NAME: `kubectl get namespace | grep ns-gpawc ` `kubectl create namespace ns-gpawc` namespace/ns-gpawc created create namespace ns-gpawc done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 120M 0 --:--:-- --:--:-- --:--:-- 120M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.1 done Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Kubernetes Env: v1.32.6 check snapshot controller check snapshot controller done POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default KubeBlocks version is:1.0.1 skip upgrade KubeBlocks current KubeBlocks version: 1.0.1 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:etcd set component version set component version:etcd set service versions:3.6.1,3.5.15,3.5.6 set service versions sorted:3.5.6,3.5.15,3.6.1 set etcd component definition set etcd component definition etcd-3-1.0.1 REPORT_COUNT 0:0 set replicas first:3,3.5.6|3,3.5.15|3,3.6.1 set replicas third:3,3.5.6 set replicas fourth:3,3.5.6 set minimum cmpv service version set minimum cmpv service version replicas:3,3.5.6 REPORT_COUNT:1 Not support cluster topology for etcd set etcd component definition set etcd component definition etcd-3-1.0.1 LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 CLUSTER_NAME:etcd-wpzwbk No resources found in ns-gpawc namespace. pod_info: termination_policy:Delete create 3 replica Delete etcd cluster check component definition set component definition by component version check cmpd by labels set component definition1: etcd-3-1.0.1 by component version:etcd apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: etcd-wpzwbk namespace: ns-gpawc spec: terminationPolicy: Delete componentSpecs: - name: etcd componentDef: etcd-3-1.0.1 tls: false replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi services: - name: client serviceName: client spec: type: NodePort ports: - port: 2379 targetPort: 2379 componentSelector: etcd roleSelector: leader `kubectl apply -f test_create_etcd-wpzwbk.yaml` cluster.apps.kubeblocks.io/etcd-wpzwbk created apply test_create_etcd-wpzwbk.yaml Success `rm -rf test_create_etcd-wpzwbk.yaml` check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Creating Sep 11,2025 17:20 UTC+0800 cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:20 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:22 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:22 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=etcd-wpzwbk` No resources found in ns-gpawc namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:2379;DB_DATABASE: There is no password in Type: 15. describe cluster `kbcli cluster describe etcd-wpzwbk --namespace ns-gpawc ` Name: etcd-wpzwbk Created Time: Sep 11,2025 17:20 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-gpawc Running Delete Endpoints: COMPONENT INTERNAL EXTERNAL Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME etcd 3.6.1 etcd-wpzwbk-etcd-0 leader Running 0 aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:20 UTC+0800 etcd 3.6.1 etcd-wpzwbk-etcd-1 follower Running 0 aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:22 UTC+0800 etcd 3.6.1 etcd-wpzwbk-etcd-2 follower Running 0 aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:22 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS etcd 100m / 100m 512Mi / 512Mi data:1Gi default Images: COMPONENT COMPONENT-DEFINITION IMAGE etcd etcd-3-1.0.1 docker.io/apecloud/etcd:v3.6.1 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-gpawc etcd-wpzwbk `kbcli cluster label etcd-wpzwbk app.kubernetes.io/instance- --namespace ns-gpawc ` label "app.kubernetes.io/instance" not found. `kbcli cluster label etcd-wpzwbk app.kubernetes.io/instance=etcd-wpzwbk --namespace ns-gpawc ` `kbcli cluster label etcd-wpzwbk --list --namespace ns-gpawc ` NAME NAMESPACE LABELS etcd-wpzwbk ns-gpawc app.kubernetes.io/instance=etcd-wpzwbk label cluster app.kubernetes.io/instance=etcd-wpzwbk Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=etcd-wpzwbk --namespace ns-gpawc ` `kbcli cluster label etcd-wpzwbk --list --namespace ns-gpawc ` NAME NAMESPACE LABELS etcd-wpzwbk ns-gpawc app.kubernetes.io/instance=etcd-wpzwbk case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label etcd-wpzwbk case.name=kbcli.test2 --overwrite --namespace ns-gpawc ` `kbcli cluster label etcd-wpzwbk --list --namespace ns-gpawc ` NAME NAMESPACE LABELS etcd-wpzwbk ns-gpawc app.kubernetes.io/instance=etcd-wpzwbk case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label etcd-wpzwbk case.name- --namespace ns-gpawc ` `kbcli cluster label etcd-wpzwbk --list --namespace ns-gpawc ` NAME NAMESPACE LABELS etcd-wpzwbk ns-gpawc app.kubernetes.io/instance=etcd-wpzwbk delete cluster label case.name Success cluster connect No resources found in ns-gpawc namespace. `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 member list" | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file 43aa49bd61cffe6b, started, etcd-wpzwbk-etcd-2, http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380, http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2379, false 493dc151d116c622, started, etcd-wpzwbk-etcd-0, http://etcd-wpzwbk-etcd-0.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380, http://etcd-wpzwbk-etcd-0.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2379, false b4e2609d3295d640, started, etcd-wpzwbk-etcd-1, http://etcd-wpzwbk-etcd-1.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380, http://etcd-wpzwbk-etcd-1.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2379, false connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-etcd-wpzwbk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-etcd-wpzwbk --namespace ns-gpawc ` Error from server (NotFound): pods "test-db-client-executionloop-etcd-wpzwbk" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-etcd-wpzwbk" not found `kubectl get secrets -l app.kubernetes.io/instance=etcd-wpzwbk` No resources found in ns-gpawc namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:2379;DB_DATABASE: No resources found in ns-gpawc namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-etcd-wpzwbk namespace: ns-gpawc spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "etcd-wpzwbk-client.ns-gpawc.svc.cluster.local" - "--user" - "" - "--password" - "" - "--port" - "2379" - "--dbtype" - "etcd" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-etcd-wpzwbk.yaml` pod/test-db-client-executionloop-etcd-wpzwbk created apply test-db-client-executionloop-etcd-wpzwbk.yaml Success `rm -rf test-db-client-executionloop-etcd-wpzwbk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 1/1 Running 0 61s check pod test-db-client-executionloop-etcd-wpzwbk status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-etcd-wpzwbk 0/1 Completed 0 66s check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:20 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:22 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:22 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done --host etcd-wpzwbk-client.ns-gpawc.svc.cluster.local --user --password --port 2379 --dbtype etcd --test executionloop --duration 60 --interval 1 Using no auth SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Execution loop start: Get key prefix:executions_loop_key Execution loop start:PUT:executions_loop_key_1:executions_loop_value_1 [ 1s ] executions total: 74 successful: 74 failed: 0 disconnect: 0 [ 2s ] executions total: 168 successful: 168 failed: 0 disconnect: 0 [ 3s ] executions total: 257 successful: 257 failed: 0 disconnect: 0 [ 4s ] executions total: 333 successful: 333 failed: 0 disconnect: 0 [ 5s ] executions total: 403 successful: 403 failed: 0 disconnect: 0 [ 6s ] executions total: 470 successful: 470 failed: 0 disconnect: 0 [ 7s ] executions total: 564 successful: 564 failed: 0 disconnect: 0 [ 8s ] executions total: 640 successful: 640 failed: 0 disconnect: 0 [ 9s ] executions total: 716 successful: 716 failed: 0 disconnect: 0 [ 10s ] executions total: 785 successful: 785 failed: 0 disconnect: 0 [ 11s ] executions total: 864 successful: 864 failed: 0 disconnect: 0 [ 12s ] executions total: 953 successful: 953 failed: 0 disconnect: 0 [ 13s ] executions total: 1035 successful: 1035 failed: 0 disconnect: 0 [ 14s ] executions total: 1113 successful: 1113 failed: 0 disconnect: 0 [ 15s ] executions total: 1206 successful: 1206 failed: 0 disconnect: 0 [ 16s ] executions total: 1294 successful: 1294 failed: 0 disconnect: 0 [ 17s ] executions total: 1363 successful: 1363 failed: 0 disconnect: 0 [ 18s ] executions total: 1435 successful: 1435 failed: 0 disconnect: 0 [ 19s ] executions total: 1509 successful: 1509 failed: 0 disconnect: 0 [ 20s ] executions total: 1587 successful: 1587 failed: 0 disconnect: 0 [ 21s ] executions total: 1664 successful: 1664 failed: 0 disconnect: 0 [ 22s ] executions total: 1727 successful: 1727 failed: 0 disconnect: 0 [ 23s ] executions total: 1795 successful: 1795 failed: 0 disconnect: 0 [ 24s ] executions total: 1875 successful: 1875 failed: 0 disconnect: 0 [ 25s ] executions total: 1941 successful: 1941 failed: 0 disconnect: 0 [ 26s ] executions total: 2012 successful: 2012 failed: 0 disconnect: 0 [ 27s ] executions total: 2098 successful: 2098 failed: 0 disconnect: 0 [ 28s ] executions total: 2179 successful: 2179 failed: 0 disconnect: 0 [ 29s ] executions total: 2274 successful: 2274 failed: 0 disconnect: 0 [ 30s ] executions total: 2356 successful: 2356 failed: 0 disconnect: 0 [ 31s ] executions total: 2450 successful: 2450 failed: 0 disconnect: 0 [ 32s ] executions total: 2539 successful: 2539 failed: 0 disconnect: 0 [ 33s ] executions total: 2634 successful: 2634 failed: 0 disconnect: 0 [ 34s ] executions total: 2735 successful: 2735 failed: 0 disconnect: 0 [ 35s ] executions total: 2826 successful: 2826 failed: 0 disconnect: 0 [ 36s ] executions total: 2920 successful: 2920 failed: 0 disconnect: 0 [ 37s ] executions total: 3011 successful: 3011 failed: 0 disconnect: 0 [ 38s ] executions total: 3112 successful: 3112 failed: 0 disconnect: 0 [ 39s ] executions total: 3200 successful: 3200 failed: 0 disconnect: 0 [ 40s ] executions total: 3300 successful: 3300 failed: 0 disconnect: 0 [ 41s ] executions total: 3395 successful: 3395 failed: 0 disconnect: 0 [ 42s ] executions total: 3486 successful: 3486 failed: 0 disconnect: 0 [ 43s ] executions total: 3581 successful: 3581 failed: 0 disconnect: 0 [ 44s ] executions total: 3683 successful: 3683 failed: 0 disconnect: 0 [ 45s ] executions total: 3767 successful: 3767 failed: 0 disconnect: 0 [ 46s ] executions total: 3858 successful: 3858 failed: 0 disconnect: 0 [ 47s ] executions total: 3950 successful: 3950 failed: 0 disconnect: 0 [ 48s ] executions total: 4043 successful: 4043 failed: 0 disconnect: 0 [ 49s ] executions total: 4136 successful: 4136 failed: 0 disconnect: 0 [ 50s ] executions total: 4218 successful: 4218 failed: 0 disconnect: 0 [ 51s ] executions total: 4301 successful: 4301 failed: 0 disconnect: 0 [ 52s ] executions total: 4371 successful: 4371 failed: 0 disconnect: 0 [ 53s ] executions total: 4459 successful: 4459 failed: 0 disconnect: 0 [ 54s ] executions total: 4536 successful: 4536 failed: 0 disconnect: 0 [ 55s ] executions total: 4631 successful: 4631 failed: 0 disconnect: 0 [ 56s ] executions total: 4718 successful: 4718 failed: 0 disconnect: 0 [ 57s ] executions total: 4810 successful: 4810 failed: 0 disconnect: 0 [ 60s ] executions total: 4894 successful: 4894 failed: 0 disconnect: 0 Test Result: Total Executions: 4894 Successful Executions: 4894 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: etcd Host: etcd-wpzwbk-client.ns-gpawc.svc.cluster.local Port: 2379 Database: Table: User: Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 4894 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-etcd-wpzwbk --namespace ns-gpawc ` pod/test-db-client-executionloop-etcd-wpzwbk patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-etcd-wpzwbk" force deleted test failover fullcpuover check cluster status before cluster-failover-fullcpuover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-etcd-wpzwbk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-etcd-wpzwbk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-etcd-wpzwbk created apply test-chaos-mesh-fullcpuover-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-fullcpuover-etcd-wpzwbk.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Failed Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:20 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:22 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:22 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-etcd-wpzwbk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-1 failover fullcpuover Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote etcd-wpzwbk --auto-approve --force=true --instance etcd-wpzwbk-etcd-1 --candidate etcd-wpzwbk-etcd-0 --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-switchover-gk2r9 created successfully, you can view the progress: kbcli cluster describe-ops etcd-wpzwbk-switchover-gk2r9 -n ns-gpawc check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running -/- Sep 11,2025 17:28 UTC+0800 check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:20 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:22 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:22 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Running 0/1 Sep 11,2025 17:28 UTC+0800 check ops status done ops_status:etcd-wpzwbk-switchover-gk2r9 ns-gpawc Switchover etcd-wpzwbk etcd-wpzwbk-etcd Succeed 1/1 Sep 11,2025 17:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations etcd-wpzwbk-switchover-gk2r9 --namespace ns-gpawc ` opsrequest.operations.kubeblocks.io/etcd-wpzwbk-switchover-gk2r9 patched `kbcli cluster delete-ops --name etcd-wpzwbk-switchover-gk2r9 --force --auto-approve --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-switchover-gk2r9 deleted No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success switchover pod:etcd-wpzwbk-etcd-0 switchover success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart etcd-wpzwbk --auto-approve --force=true --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-restart-nr28s created successfully, you can view the progress: kbcli cluster describe-ops etcd-wpzwbk-restart-nr28s -n ns-gpawc check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-restart-nr28s ns-gpawc Restart etcd-wpzwbk etcd Pending -/- Sep 11,2025 17:30 UTC+0800 check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Updating Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:31 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash` check cluster connect done check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-restart-nr28s ns-gpawc Restart etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:30 UTC+0800 check ops status done ops_status:etcd-wpzwbk-restart-nr28s ns-gpawc Restart etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations etcd-wpzwbk-restart-nr28s --namespace ns-gpawc ` opsrequest.operations.kubeblocks.io/etcd-wpzwbk-restart-nr28s patched `kbcli cluster delete-ops --name etcd-wpzwbk-restart-nr28s --force --auto-approve --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-restart-nr28s deleted No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=etcd-wpzwbk,apps.kubeblocks.io/component-name=etcd,apps.kubeblocks.io/vct-name=data --namespace ns-gpawc ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in etcd-wpzwbk namespace. `kbcli cluster volume-expand etcd-wpzwbk --auto-approve --force=true --components etcd --volume-claim-templates data --storage 3Gi --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-volumeexpansion-8kz55 created successfully, you can view the progress: kbcli cluster describe-ops etcd-wpzwbk-volumeexpansion-8kz55 -n ns-gpawc check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-volumeexpansion-8kz55 ns-gpawc VolumeExpansion etcd-wpzwbk etcd Running 0/3 Sep 11,2025 17:33 UTC+0800 check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Updating Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:31 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash` check cluster connect done No resources found in etcd-wpzwbk namespace. check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-volumeexpansion-8kz55 ns-gpawc VolumeExpansion etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:33 UTC+0800 check ops status done ops_status:etcd-wpzwbk-volumeexpansion-8kz55 ns-gpawc VolumeExpansion etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations etcd-wpzwbk-volumeexpansion-8kz55 --namespace ns-gpawc ` opsrequest.operations.kubeblocks.io/etcd-wpzwbk-volumeexpansion-8kz55 patched `kbcli cluster delete-ops --name etcd-wpzwbk-volumeexpansion-8kz55 --force --auto-approve --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-volumeexpansion-8kz55 deleted No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-etcd-wpzwbk" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-2 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-etcd-wpzwbk.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-etcd-wpzwbk created apply test-chaos-mesh-podfailure-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-podfailure-etcd-wpzwbk.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Failed Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:31 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-etcd-wpzwbk" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-1 failover podfailure Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-etcd-wpzwbk" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-1 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-etcd-wpzwbk.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-etcd-wpzwbk created apply test-chaos-mesh-podkill-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-podkill-etcd-wpzwbk.yaml` check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Updating Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:42 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-etcd-wpzwbk" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-etcd-wpzwbk patched check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover podkill Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-etcd-wpzwbk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-etcd-wpzwbk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-etcd-wpzwbk created apply test-chaos-mesh-networkbandwidthover-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-etcd-wpzwbk.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:42 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-etcd-wpzwbk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-2 failover networkbandwidthover Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-2 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-etcd-wpzwbk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-etcd-wpzwbk created apply test-chaos-mesh-oom-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-oom-etcd-wpzwbk.yaml` check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:42 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-etcd-wpzwbk patched check failover pod name failover pod name:etcd-wpzwbk-etcd-2 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-2 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-etcd-wpzwbk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-etcd-wpzwbk created apply test-chaos-mesh-oom-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-oom-etcd-wpzwbk.yaml` check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Failed Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:42 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-etcd-wpzwbk" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-etcd-wpzwbk patched check failover pod name:etcd-wpzwbk-etcd-1 failover oom Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-etcd-wpzwbk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-1 action: partition mode: all target: mode: all selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-etcd-wpzwbk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-etcd-wpzwbk created apply test-chaos-mesh-networkpartition-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-networkpartition-etcd-wpzwbk.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:42 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-etcd-wpzwbk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-1 failover networkpartition Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-etcd-wpzwbk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-etcd-wpzwbk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-etcd-wpzwbk created apply test-chaos-mesh-networklossover-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-networklossover-etcd-wpzwbk.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:32 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:42 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:30 UTC+0800 check pod status done check cluster role leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-etcd-wpzwbk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover networklossover Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop etcd-wpzwbk --auto-approve --force=true --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-stop-wr4tt created successfully, you can view the progress: kbcli cluster describe-ops etcd-wpzwbk-stop-wr4tt -n ns-gpawc check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-stop-wr4tt ns-gpawc Stop etcd-wpzwbk Running -/- Sep 11,2025 17:52 UTC+0800 check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Stopped Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-stop-wr4tt ns-gpawc Stop etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:52 UTC+0800 check ops status done ops_status:etcd-wpzwbk-stop-wr4tt ns-gpawc Stop etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations etcd-wpzwbk-stop-wr4tt --namespace ns-gpawc ` opsrequest.operations.kubeblocks.io/etcd-wpzwbk-stop-wr4tt patched `kbcli cluster delete-ops --name etcd-wpzwbk-stop-wr4tt --force --auto-approve --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-stop-wr4tt deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start etcd-wpzwbk --force=true --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-start-sztkj created successfully, you can view the progress: kbcli cluster describe-ops etcd-wpzwbk-start-sztkj -n ns-gpawc check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-start-sztkj ns-gpawc Start etcd-wpzwbk etcd Running 0/3 Sep 11,2025 17:52 UTC+0800 check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Updating Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-start-sztkj ns-gpawc Start etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:52 UTC+0800 check ops status done ops_status:etcd-wpzwbk-start-sztkj ns-gpawc Start etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 17:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations etcd-wpzwbk-start-sztkj --namespace ns-gpawc ` opsrequest.operations.kubeblocks.io/etcd-wpzwbk-start-sztkj patched `kbcli cluster delete-ops --name etcd-wpzwbk-start-sztkj --force --auto-approve --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-start-sztkj deleted No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running `kill 1` Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done check failover pod name failover pod name:etcd-wpzwbk-etcd-0 checking failover... `kill 1` Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-2 check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done check failover pod name:etcd-wpzwbk-etcd-1 failover kill1 Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-etcd-wpzwbk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-etcd-wpzwbk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-etcd-wpzwbk created apply test-chaos-mesh-dnsrandom-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-dnsrandom-etcd-wpzwbk.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-1;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-etcd-wpzwbk" force deleted dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-etcd-wpzwbk patched check failover pod name failover pod name:etcd-wpzwbk-etcd-1 failover dnsrandom Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover networkdelayover check cluster status before cluster-failover-networkdelayover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelayover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-etcd-wpzwbk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelayover-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelayover-etcd-wpzwbk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelayover-etcd-wpzwbk created apply test-chaos-mesh-networkdelayover-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-networkdelayover-etcd-wpzwbk.yaml` networkdelayover chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelayover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-etcd-wpzwbk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover networkdelayover Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-etcd-wpzwbk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-etcd-wpzwbk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-etcd-wpzwbk created apply test-chaos-mesh-networkduplicate-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-networkduplicate-etcd-wpzwbk.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-etcd-wpzwbk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover networkduplicate Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running Error from server (NotFound): pods "test-db-client-connectionstress-etcd-wpzwbk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-etcd-wpzwbk --namespace ns-gpawc ` Error from server (NotFound): pods "test-db-client-connectionstress-etcd-wpzwbk" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-etcd-wpzwbk" not found `kubectl get secrets -l app.kubernetes.io/instance=etcd-wpzwbk` No resources found in ns-gpawc namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:2379;DB_DATABASE: No resources found in ns-gpawc namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-etcd-wpzwbk namespace: ns-gpawc spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "etcd-wpzwbk-client.ns-gpawc.svc.cluster.local" - "--user" - "" - "--password" - "" - "--port" - "2379" - "--database" - "" - "--dbtype" - "etcd" - "--test" - "connectionstress" - "--connections" - "5000" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-etcd-wpzwbk.yaml` pod/test-db-client-connectionstress-etcd-wpzwbk created apply test-db-client-connectionstress-etcd-wpzwbk.yaml Success `rm -rf test-db-client-connectionstress-etcd-wpzwbk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 9s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 61s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 66s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 71s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 76s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 82s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 87s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 92s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 97s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 102s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 107s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 112s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 1/1 Running 0 118s check pod test-db-client-connectionstress-etcd-wpzwbk status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-etcd-wpzwbk 0/1 Completed 0 2m3s check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done Connection 4958: Get operation successful Using no auth Connection 4959: Get operation successful Using no auth Connection 4960: Get operation successful Using no auth Connection 4961: Get operation successful Using no auth Connection 4962: Get operation successful Using no auth Connection 4963: Get operation successful Using no auth Connection 4964: Get operation successful Using no auth Connection 4965: Get operation successful Using no auth Connection 4966: Get operation successful Using no auth Connection 4967: Get operation successful Using no auth Connection 4968: Get operation successful Using no auth Connection 4969: Get operation successful Using no auth Connection 4970: Get operation successful Using no auth Connection 4971: Get operation successful Using no auth Connection 4972: Get operation successful Using no auth Connection 4973: Get operation successful Using no auth Connection 4974: Get operation successful Using no auth Connection 4975: Get operation successful Using no auth Connection 4976: Get operation successful Using no auth Connection 4977: Get operation successful Using no auth Connection 4978: Get operation successful Using no auth Connection 4979: Get operation successful Using no auth Connection 4980: Get operation successful Using no auth Connection 4981: Get operation successful Using no auth Connection 4982: Get operation successful Using no auth Connection 4983: Get operation successful Using no auth Connection 4984: Get operation successful Using no auth Connection 4985: Get operation successful Using no auth Connection 4986: Get operation successful Using no auth Connection 4987: Get operation successful Using no auth Connection 4988: Get operation successful Using no auth Connection 4989: Get operation successful Using no auth Connection 4990: Get operation successful Using no auth Connection 4991: Get operation successful Using no auth Connection 4992: Get operation successful Using no auth Connection 4993: Get operation successful Using no auth Connection 4994: Get operation successful Using no auth Connection 4995: Get operation successful Using no auth Connection 4996: Get operation successful Using no auth Connection 4997: Get operation successful Using no auth Connection 4998: Get operation successful Using no auth Connection 4999: Get operation successful Test Result: Connection stress test completed: Successful connections: 5000 Duration: 56 seconds Connection Information: Database Type: etcd Host: etcd-wpzwbk-client.ns-gpawc.svc.cluster.local Port: 2379 Database: Table: User: Org: Access Mode: mysql Test Type: connectionstress Connection Count: 5000 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-etcd-wpzwbk --namespace ns-gpawc ` pod/test-db-client-connectionstress-etcd-wpzwbk patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-etcd-wpzwbk" force deleted check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover connectionstress Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-etcd-wpzwbk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-etcd-wpzwbk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-etcd-wpzwbk created apply test-chaos-mesh-dnserror-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-dnserror-etcd-wpzwbk.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-etcd-wpzwbk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover dnserror Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-etcd-wpzwbk" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-etcd-wpzwbk.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-etcd-wpzwbk created apply test-chaos-mesh-timeoffset-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-timeoffset-etcd-wpzwbk.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-etcd-wpzwbk" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-0 failover timeoffset Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success skip cluster HorizontalScaling Out test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-etcd-wpzwbk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-etcd-wpzwbk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-etcd-wpzwbk namespace: ns-gpawc spec: selector: namespaces: - ns-gpawc labelSelectors: apps.kubeblocks.io/pod-name: etcd-wpzwbk-etcd-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-etcd-wpzwbk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-etcd-wpzwbk created apply test-chaos-mesh-networkcorruptover-etcd-wpzwbk.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-etcd-wpzwbk.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 17:52 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 17:54 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running leader 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 17:54 UTC+0800 check pod status done check cluster role leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 leader: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-1 check cluster role done leader: etcd-wpzwbk-etcd-2;follower: etcd-wpzwbk-etcd-0 etcd-wpzwbk-etcd-1 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-etcd-wpzwbk --namespace ns-gpawc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-etcd-wpzwbk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-etcd-wpzwbk" not found check failover pod name failover pod name:etcd-wpzwbk-etcd-2 failover networkcorruptover Success No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-2 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale etcd-wpzwbk --auto-approve --force=true --components etcd --cpu 200m --memory 0.6Gi --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-verticalscaling-d98rs created successfully, you can view the progress: kbcli cluster describe-ops etcd-wpzwbk-verticalscaling-d98rs -n ns-gpawc check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-verticalscaling-d98rs ns-gpawc VerticalScaling etcd-wpzwbk etcd Running -/- Sep 11,2025 18:12 UTC+0800 check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc Delete Updating Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 18:13 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 18:12 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 18:14 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done check ops status `kbcli cluster list-ops etcd-wpzwbk --status all --namespace ns-gpawc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME etcd-wpzwbk-verticalscaling-d98rs ns-gpawc VerticalScaling etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 18:12 UTC+0800 check ops status done ops_status:etcd-wpzwbk-verticalscaling-d98rs ns-gpawc VerticalScaling etcd-wpzwbk etcd Succeed 3/3 Sep 11,2025 18:12 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations etcd-wpzwbk-verticalscaling-d98rs --namespace ns-gpawc ` opsrequest.operations.kubeblocks.io/etcd-wpzwbk-verticalscaling-d98rs patched `kbcli cluster delete-ops --name etcd-wpzwbk-verticalscaling-d98rs --force --auto-approve --namespace ns-gpawc ` OpsRequest etcd-wpzwbk-verticalscaling-d98rs deleted No resources found in ns-gpawc namespace. check db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash ` check db_client batch data Success No resources found in ns-gpawc namespace. check readonly db_client batch data count `echo "etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 get --prefix \"executions_loop_key\" --keys-only | (grep executions_loop_key || true) | wc -l " | kubectl exec -it etcd-wpzwbk-etcd-1 --namespace ns-gpawc -- bash ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update etcd-wpzwbk --termination-policy=WipeOut --namespace ns-gpawc ` cluster.apps.kubeblocks.io/etcd-wpzwbk updated check cluster status `kbcli cluster list etcd-wpzwbk --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk ns-gpawc WipeOut Running Sep 11,2025 17:20 UTC+0800 app.kubernetes.io/instance=etcd-wpzwbk check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-etcd-0 ns-gpawc etcd-wpzwbk etcd Running leader 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 18:13 UTC+0800 etcd-wpzwbk-etcd-1 ns-gpawc etcd-wpzwbk etcd Running follower 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 18:12 UTC+0800 etcd-wpzwbk-etcd-2 ns-gpawc etcd-wpzwbk etcd Running follower 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 18:14 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-etcd-0;follower: etcd-wpzwbk-etcd-1 etcd-wpzwbk-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done cluster datafile backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-hq2nh -n kb-ubhmh -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-hq2nh -n kb-ubhmh -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-ubhmh get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-ubhmh -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-ubhmh -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-ubhmh.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-ubhmh.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-9f9cdcc54-p6xn9 --namespace kb-ubhmh -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup etcd-wpzwbk --method datafile --namespace ns-gpawc ` Backup backup-ns-gpawc-etcd-wpzwbk-20250911181617 created successfully, you can view the progress: kbcli cluster list-backups --names=backup-ns-gpawc-etcd-wpzwbk-20250911181617 -n ns-gpawc check backup status `kbcli cluster list-backups etcd-wpzwbk --namespace ns-gpawc ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-gpawc-etcd-wpzwbk-20250911181617 ns-gpawc etcd-wpzwbk datafile Delete Sep 11,2025 18:16 UTC+0800 backup_status:etcd-wpzwbk-datafile-Running backup_status:etcd-wpzwbk-datafile-Running backup_status:etcd-wpzwbk-datafile-Running check backup status done backup_status:backup-ns-gpawc-etcd-wpzwbk-20250911181617 ns-gpawc etcd-wpzwbk datafile Completed 58234 11s Delete Sep 11,2025 18:16 UTC+0800 Sep 11,2025 18:16 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "etcd-wpzwbk-backup" not found `kbcli cluster describe-backup --names backup-ns-gpawc-etcd-wpzwbk-20250911181617 --namespace ns-gpawc ` Name: backup-ns-gpawc-etcd-wpzwbk-20250911181617 Cluster: etcd-wpzwbk Namespace: ns-gpawc Spec: Method: datafile Policy Name: etcd-wpzwbk-etcd-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-gpawc-etcd-wpzwbk-20250911181617-a3c19d54 TargetPodName: etcd-wpzwbk-etcd-0 Phase: Completed Start Time: Sep 11,2025 18:16 UTC+0800 Completion Time: Sep 11,2025 18:16 UTC+0800 Status: Phase: Completed Total Size: 58234 ActionSet Name: etcdctl-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: Sep 11,2025 18:16 UTC+0800 Completion Time: Sep 11,2025 18:16 UTC+0800 Path: /ns-gpawc/etcd-wpzwbk-1b1f475d-3d0a-4695-b91b-9f57fa5b84f7/etcd/backup-ns-gpawc-etcd-wpzwbk-20250911181617 Warning Events: `kbcli cluster restore etcd-wpzwbk-backup --backup backup-ns-gpawc-etcd-wpzwbk-20250911181617 --namespace ns-gpawc ` Cluster etcd-wpzwbk-backup created check cluster status `kbcli cluster list etcd-wpzwbk-backup --show-labels --namespace ns-gpawc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS etcd-wpzwbk-backup ns-gpawc WipeOut Sep 11,2025 18:16 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances etcd-wpzwbk-backup --namespace ns-gpawc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME etcd-wpzwbk-backup-etcd-0 ns-gpawc etcd-wpzwbk-backup etcd Running leader 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000000/10.224.0.5 Sep 11,2025 18:17 UTC+0800 etcd-wpzwbk-backup-etcd-1 ns-gpawc etcd-wpzwbk-backup etcd Running follower 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000002/10.224.0.9 Sep 11,2025 18:18 UTC+0800 etcd-wpzwbk-backup-etcd-2 ns-gpawc etcd-wpzwbk-backup etcd Running follower 0 200m / 200m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-42771698-vmss000004/10.224.0.6 Sep 11,2025 18:18 UTC+0800 check pod status done check cluster role check cluster role done leader: etcd-wpzwbk-backup-etcd-0;follower: etcd-wpzwbk-backup-etcd-1 etcd-wpzwbk-backup-etcd-2 No resources found in ns-gpawc namespace. check cluster connect `echo 'etcdctl --endpoints=http://etcd-wpzwbk-backup-client.ns-gpawc.svc.cluster.local:2379 endpoint health' | kubectl exec -it etcd-wpzwbk-backup-etcd-0 --namespace ns-gpawc -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-gpawc-etcd-wpzwbk-20250911181617 --namespace ns-gpawc ` Name: backup-ns-gpawc-etcd-wpzwbk-20250911181617 Cluster: etcd-wpzwbk Namespace: ns-gpawc Spec: Method: datafile Policy Name: etcd-wpzwbk-etcd-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-gpawc-etcd-wpzwbk-20250911181617-a3c19d54 TargetPodName: etcd-wpzwbk-etcd-0 Phase: Completed Start Time: Sep 11,2025 18:16 UTC+0800 Completion Time: Sep 11,2025 18:16 UTC+0800 Status: Phase: Completed Total Size: 58234 ActionSet Name: etcdctl-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: Sep 11,2025 18:16 UTC+0800 Completion Time: Sep 11,2025 18:16 UTC+0800 Path: /ns-gpawc/etcd-wpzwbk-1b1f475d-3d0a-4695-b91b-9f57fa5b84f7/etcd/backup-ns-gpawc-etcd-wpzwbk-20250911181617 Warning Events: cluster connect No resources found in ns-gpawc namespace. `echo "etcdctl --endpoints=http://etcd-wpzwbk-backup-client.ns-gpawc.svc.cluster.local:2379 member list" | kubectl exec -it etcd-wpzwbk-backup-etcd-0 --namespace ns-gpawc -- bash ` Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file 62da3f1a5fdc91f, started, etcd-wpzwbk-backup-etcd-0, http://etcd-wpzwbk-backup-etcd-0.etcd-wpzwbk-backup-etcd-headless.ns-gpawc.svc.cluster.local:2380, http://etcd-wpzwbk-backup-etcd-0.etcd-wpzwbk-backup-etcd-headless.ns-gpawc.svc.cluster.local:2379, false 90421002defba9d1, started, etcd-wpzwbk-backup-etcd-1, http://etcd-wpzwbk-backup-etcd-1.etcd-wpzwbk-backup-etcd-headless.ns-gpawc.svc.cluster.local:2380, http://etcd-wpzwbk-backup-etcd-1.etcd-wpzwbk-backup-etcd-headless.ns-gpawc.svc.cluster.local:2379, false aa2fc8d1421417c6, started, etcd-wpzwbk-backup-etcd-2, http://etcd-wpzwbk-backup-etcd-2.etcd-wpzwbk-backup-etcd-headless.ns-gpawc.svc.cluster.local:2380, http://etcd-wpzwbk-backup-etcd-2.etcd-wpzwbk-backup-etcd-headless.ns-gpawc.svc.cluster.local:2379, false connect cluster Success delete cluster etcd-wpzwbk-backup `kbcli cluster delete etcd-wpzwbk-backup --auto-approve --namespace ns-gpawc ` Cluster etcd-wpzwbk-backup deleted pod_info:etcd-wpzwbk-backup-etcd-0 2/2 Running 0 2m49s etcd-wpzwbk-backup-etcd-1 2/2 Running 0 112s etcd-wpzwbk-backup-etcd-2 2/2 Running 0 86s No resources found in ns-gpawc namespace. delete cluster pod done No resources found in ns-gpawc namespace. check cluster resource non-exist OK: pvc No resources found in ns-gpawc namespace. delete cluster done No resources found in ns-gpawc namespace. No resources found in ns-gpawc namespace. No resources found in ns-gpawc namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-gpawc-etcd-wpzwbk-20250911181617 --namespace ns-gpawc ` backup.dataprotection.kubeblocks.io/backup-ns-gpawc-etcd-wpzwbk-20250911181617 patched `kbcli cluster delete-backup etcd-wpzwbk --name backup-ns-gpawc-etcd-wpzwbk-20250911181617 --force --auto-approve --namespace ns-gpawc ` Backup backup-ns-gpawc-etcd-wpzwbk-20250911181617 deleted No opsrequests found in ns-gpawc namespace. cluster list-logs `kbcli cluster list-logs etcd-wpzwbk --namespace ns-gpawc ` No log files found. Error from server (NotFound): pods "etcd-wpzwbk-etcd-0" not found cluster logs `kbcli cluster logs etcd-wpzwbk --tail 30 --namespace ns-gpawc ` Defaulted container "etcd" out of: etcd, kbagent, inject-bash (init), init-kbagent (init), kbagent-worker (init) ***"level":"warn","ts":"2025-09-11T10:14:56.817671Z","caller":"rafthttp/peer.go:254","msg":"dropped internal Raft message since sending buffer is full","message-type":"MsgHeartbeat","local-member-id":"493dc151d116c622","from":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b","remote-peer-name":"pipeline","remote-peer-active":false*** ***"level":"warn","ts":"2025-09-11T10:14:56.917252Z","caller":"rafthttp/peer.go:254","msg":"dropped internal Raft message since sending buffer is full","message-type":"MsgHeartbeat","local-member-id":"493dc151d116c622","from":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b","remote-peer-name":"pipeline","remote-peer-active":false*** ***"level":"warn","ts":"2025-09-11T10:14:57.017507Z","caller":"rafthttp/peer.go:254","msg":"dropped internal Raft message since sending buffer is full","message-type":"MsgHeartbeat","local-member-id":"493dc151d116c622","from":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b","remote-peer-name":"pipeline","remote-peer-active":false*** ***"level":"warn","ts":"2025-09-11T10:14:57.116710Z","caller":"rafthttp/peer.go:254","msg":"dropped internal Raft message since sending buffer is full","message-type":"MsgHeartbeat","local-member-id":"493dc151d116c622","from":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b","remote-peer-name":"pipeline","remote-peer-active":false*** ***"level":"info","ts":"2025-09-11T10:14:57.299249Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"493dc151d116c622","to":"43aa49bd61cffe6b","stream-type":"stream Message"*** ***"level":"info","ts":"2025-09-11T10:14:57.299290Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"43aa49bd61cffe6b"*** ***"level":"info","ts":"2025-09-11T10:14:57.299297Z","caller":"rafthttp/stream.go:248","msg":"set message encoder","from":"493dc151d116c622","to":"43aa49bd61cffe6b","stream-type":"stream MsgApp v2"*** ***"level":"info","ts":"2025-09-11T10:14:57.299325Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream MsgApp v2","local-member-id":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b"*** ***"level":"info","ts":"2025-09-11T10:14:57.299317Z","caller":"rafthttp/stream.go:273","msg":"established TCP streaming connection with remote peer","stream-writer-type":"stream Message","local-member-id":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b"*** ***"level":"warn","ts":"2025-09-11T10:14:57.318951Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"43aa49bd61cffe6b","error":"failed to write 43aa49bd61cffe6b on pipeline (dial tcp 10.244.1.87:2380: i/o timeout)"*** ***"level":"warn","ts":"2025-09-11T10:14:57.538025Z","caller":"etcdserver/cluster_util.go:259","msg":"failed to reach the peer URL","address":"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version","remote-member-id":"43aa49bd61cffe6b","error":"Get \"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version\": dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:14:57.538081Z","caller":"etcdserver/cluster_util.go:160","msg":"failed to get version","remote-member-id":"43aa49bd61cffe6b","error":"Get \"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version\": dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:00.732476Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"43aa49bd61cffe6b","rtt":"2.821099ms","error":"dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:00.732497Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"43aa49bd61cffe6b","rtt":"391.567µs","error":"dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:03.543392Z","caller":"etcdserver/cluster_util.go:259","msg":"failed to reach the peer URL","address":"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version","remote-member-id":"43aa49bd61cffe6b","error":"Get \"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version\": dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:03.543459Z","caller":"etcdserver/cluster_util.go:160","msg":"failed to get version","remote-member-id":"43aa49bd61cffe6b","error":"Get \"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version\": dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:05.733368Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_RAFT_MESSAGE","remote-peer-id":"43aa49bd61cffe6b","rtt":"2.821099ms","error":"dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:05.733456Z","caller":"rafthttp/probing_status.go:68","msg":"prober detected unhealthy status","round-tripper-name":"ROUND_TRIPPER_SNAPSHOT","remote-peer-id":"43aa49bd61cffe6b","rtt":"391.567µs","error":"dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"info","ts":"2025-09-11T10:15:08.321171Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"43aa49bd61cffe6b"*** ***"level":"info","ts":"2025-09-11T10:15:08.321213Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream MsgApp v2","local-member-id":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b"*** ***"level":"warn","ts":"2025-09-11T10:15:09.263853Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"43aa49bd61cffe6b","error":"failed to write 43aa49bd61cffe6b on pipeline (dial tcp 10.244.1.87:2380: i/o timeout)"*** ***"level":"warn","ts":"2025-09-11T10:15:09.545078Z","caller":"etcdserver/cluster_util.go:259","msg":"failed to reach the peer URL","address":"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version","remote-member-id":"43aa49bd61cffe6b","error":"Get \"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version\": dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"warn","ts":"2025-09-11T10:15:09.545156Z","caller":"etcdserver/cluster_util.go:160","msg":"failed to get version","remote-member-id":"43aa49bd61cffe6b","error":"Get \"http://etcd-wpzwbk-etcd-2.etcd-wpzwbk-etcd-headless.ns-gpawc.svc.cluster.local:2380/version\": dial tcp 10.244.1.87:2380: i/o timeout"*** ***"level":"info","ts":"2025-09-11T10:15:09.732407Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"43aa49bd61cffe6b"*** ***"level":"warn","ts":"2025-09-11T10:15:10.299471Z","caller":"rafthttp/peer_status.go:66","msg":"peer became inactive (message send to peer failed)","peer-id":"43aa49bd61cffe6b","error":"failed to dial 43aa49bd61cffe6b on stream Message (dial tcp 10.244.1.87:2380: i/o timeout)"*** ***"level":"info","ts":"2025-09-11T10:15:10.302862Z","caller":"rafthttp/peer_status.go:53","msg":"peer became active","peer-id":"43aa49bd61cffe6b"*** ***"level":"info","ts":"2025-09-11T10:15:10.302899Z","caller":"rafthttp/stream.go:411","msg":"established TCP streaming connection with remote peer","stream-reader-type":"stream Message","local-member-id":"493dc151d116c622","remote-peer-id":"43aa49bd61cffe6b"*** ***"level":"info","ts":"2025-09-11T10:16:19.752365Z","caller":"v3rpc/maintenance.go:153","msg":"sending database snapshot to client","total-bytes":565248,"size":"565 kB","storage-version":"3.6.0"*** ***"level":"info","ts":"2025-09-11T10:16:19.758328Z","caller":"v3rpc/maintenance.go:195","msg":"sending database sha256 checksum to client","total-bytes":565248,"checksum-size":32*** ***"level":"info","ts":"2025-09-11T10:16:19.758396Z","caller":"v3rpc/maintenance.go:204","msg":"successfully sent database snapshot to client","total-bytes":565248,"size":"565 kB","took":"6.030145ms","storage-version":"3.6.0"*** delete cluster etcd-wpzwbk `kbcli cluster delete etcd-wpzwbk --auto-approve --namespace ns-gpawc ` Cluster etcd-wpzwbk deleted pod_info:etcd-wpzwbk-etcd-0 2/2 Running 0 6m43s etcd-wpzwbk-etcd-1 2/2 Running 0 7m55s etcd-wpzwbk-etcd-2 2/2 Running 0 5m35s No resources found in ns-gpawc namespace. delete cluster pod done No resources found in ns-gpawc namespace. check cluster resource non-exist OK: pvc No resources found in ns-gpawc namespace. delete cluster done No resources found in ns-gpawc namespace. No resources found in ns-gpawc namespace. No resources found in ns-gpawc namespace. Etcd Test Suite All Done! Test Engine: etcd Test Type: 15 --------------------------------------Etcd (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=etcd-3-1.0.1;ComponentVersion=etcd;ServiceVersion=3.5.6;]|[Description=Create a cluster with the specified component definition etcd-3-1.0.1 and component version etcd and service version 3.5.6] [PASSED]|[Connect]|[ComponentName=etcd]|[Description=Connect to the cluster] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=etcd]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[SwitchOver]|[ComponentName=etcd]|[Description=SwitchOver the cluster specify component etcd] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=etcd]|[Description=VolumeExpansion the cluster specify component etcd] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=etcd]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Pod Kill;ComponentName=etcd]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=etcd]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=etcd]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=etcd]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=etcd]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=etcd]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=etcd]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=etcd]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=etcd]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=etcd]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=etcd]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=etcd]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=etcd]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[VerticalScaling]|[ComponentName=etcd]|[Description=VerticalScaling the cluster specify component etcd] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=datafile]|[Description=The cluster datafile Backup] [PASSED]|[Restore]|[BackupMethod=datafile]|[Description=The cluster datafile Restore] [PASSED]|[Connect]|[ComponentName=etcd]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=datafile]|[Description=Delete the datafile restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]