source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-svryh ` `kubectl create namespace ns-svryh` namespace/ns-svryh created create namespace ns-svryh done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 93 32.1M 93 30.0M 0 0 121M 0 --:--:-- --:--:-- --:--:-- 121M 100 32.1M 100 32.1M 0 0 121M 0 --:--:-- --:--:-- --:--:-- 123M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:kingbase set component version set component version:kingbase-cluster set service versions:8.6.8 set service versions sorted:8.6.8 no cluster version found set kingbase component definition set kingbase component definition kingbase set replicas first:3,8.6.8 set replicas third:3,8.6.8 set replicas fourth:3,8.6.8 set minimum cmpv service version set minimum cmpv service version replicas:3,8.6.8 REPORT_COUNT:1 CLUSTER_TOPOLOGY:kingbase-cluster topology kingbase-cluster found in cluster definition kingbase LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-svryh namespace. termination_policy:DoNotTerminate create 3 replica DoNotTerminate kingbase cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition2: kingbase-monitor by component version:kingbase-cluster apiVersion: v1 kind: Secret metadata: name: kingbase-jzvvix-license namespace: ns-svryh type: Opaque --- apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: kingbase-jzvvix namespace: ns-svryh spec: clusterDefinitionRef: kingbase topology: kingbase-cluster terminationPolicy: DoNotTerminate componentSpecs: - name: kingbase serviceVersion: 8.6.8 schedulingPolicy: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app.kubernetes.io/instance: kingbase-jzvvix apps.kubeblocks.io/component-name: kingbase topologyKey: kubernetes.io/hostname replicas: 3 volumes: - name: license secret: secretName: kingbase-jzvvix-license resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: etc spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: kmonitor serviceVersion: 8.6.8 schedulingPolicy: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app.kubernetes.io/instance: kingbase-jzvvix apps.kubeblocks.io/component-name: kmonitor topologyKey: kubernetes.io/hostname replicas: 1 resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: etc spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_kingbase-jzvvix.yaml` secret/kingbase-jzvvix-license created cluster.apps.kubeblocks.io/kingbase-jzvvix created apply test_create_kingbase-jzvvix.yaml Success `rm -rf test_create_kingbase-jzvvix.yaml` check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase DoNotTerminate Jun 19,2025 18:16 UTC+0800 clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:17 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0;standby: kingbase-jzvvix-kingbase-1 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check pod kingbase-jzvvix-kingbase-0 container_name kingbase exist password Kingbaseha110 check pod kingbase-jzvvix-kingbase-0 container_name lorry exist password Kingbaseha110 check pod kingbase-jzvvix-kingbase-0 container_name config-manager exist password Kingbaseha110 No container logs contain secret password. describe cluster `kbcli cluster describe kingbase-jzvvix --namespace ns-svryh ` Name: kingbase-jzvvix Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-svryh kingbase Running DoNotTerminate Endpoints: COMPONENT MODE INTERNAL EXTERNAL kingbase ReadWrite kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local:54321 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME kingbase kingbase-jzvvix-kingbase-0 primary Running us-west-2a ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:17 UTC+0800 kingbase kingbase-jzvvix-kingbase-1 standby Running us-west-2a ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 kingbase kingbase-jzvvix-kingbase-2 standby Running us-west-2a ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 kmonitor kingbase-jzvvix-kmonitor-0 Running us-west-2a ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS kingbase false 500m / 500m 512Mi / 512Mi data:1Gi kb-default-sc etc:1Gi kb-default-sc kmonitor false 500m / 500m 512Mi / 512Mi data:1Gi kb-default-sc etc:1Gi kb-default-sc Images: COMPONENT TYPE IMAGE kingbase docker.io/apecloud/kingbase:v008r006c008b0014 kmonitor docker.io/apecloud/kingbase-monitor:v2.4.0 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME backuprepo-kbcli-test Enabled 0 2 */1 * * full 7d backuprepo-kbcli-test Enabled 0 2 */1 * * full 7d Show cluster events: kbcli cluster list-events -n ns-svryh kingbase-jzvvix `kbcli cluster label kingbase-jzvvix app.kubernetes.io/instance- --namespace ns-svryh ` label "app.kubernetes.io/instance" not found. `kbcli cluster label kingbase-jzvvix app.kubernetes.io/instance=kingbase-jzvvix --namespace ns-svryh ` `kbcli cluster label kingbase-jzvvix --list --namespace ns-svryh ` NAME NAMESPACE LABELS kingbase-jzvvix ns-svryh app.kubernetes.io/instance=kingbase-jzvvix clusterdefinition.kubeblocks.io/name=kingbase clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=kingbase-jzvvix Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=kingbase-jzvvix --namespace ns-svryh ` `kbcli cluster label kingbase-jzvvix --list --namespace ns-svryh ` NAME NAMESPACE LABELS kingbase-jzvvix ns-svryh app.kubernetes.io/instance=kingbase-jzvvix case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=kingbase clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label kingbase-jzvvix case.name=kbcli.test2 --overwrite --namespace ns-svryh ` `kbcli cluster label kingbase-jzvvix --list --namespace ns-svryh ` NAME NAMESPACE LABELS kingbase-jzvvix ns-svryh app.kubernetes.io/instance=kingbase-jzvvix case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=kingbase clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label kingbase-jzvvix case.name- --namespace ns-svryh ` `kbcli cluster label kingbase-jzvvix --list --namespace ns-svryh ` NAME NAMESPACE LABELS kingbase-jzvvix ns-svryh app.kubernetes.io/instance=kingbase-jzvvix clusterdefinition.kubeblocks.io/name=kingbase clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. `echo "echo \"show max_connections;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash ` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file max_connections ----------------- 100 (1 row) connect cluster Success set max_connections to 100 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-kingbase-jzvvix" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-kingbase-jzvvix --namespace ns-svryh ` Error from server (NotFound): pods "test-db-client-executionloop-kingbase-jzvvix" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-kingbase-jzvvix" not found `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-kingbase-jzvvix namespace: ns-svryh spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local" - "--user" - "esrep" - "--password" - "Kingbaseha110" - "--port" - "54321" - "--dbtype" - "kingbase" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-kingbase-jzvvix.yaml` pod/test-db-client-executionloop-kingbase-jzvvix created apply test-db-client-executionloop-kingbase-jzvvix.yaml Success `rm -rf test-db-client-executionloop-kingbase-jzvvix.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 12s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 18s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 24s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 36s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 42s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 48s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 54s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 1/1 Running 0 60s check pod test-db-client-executionloop-kingbase-jzvvix status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kingbase-jzvvix 0/1 Completed 0 66s check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase DoNotTerminate Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:17 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0;standby: kingbase-jzvvix-kingbase-1 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done --host kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local --user esrep --password Kingbaseha110 --port 54321 --dbtype kingbase --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:24:48.260 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: create databases executions_loop CREATE DATABASE executions_loop; reconnect connection executions_loop drop table executions_loop_table DROP TABLE IF EXISTS executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop_table (id SERIAL PRIMARY KEY , value text); Execution loop start:INSERT INTO executions_loop_table (value) VALUES ('executions_loop_test_1'); [ 1s ] executions total: 92 successful: 92 failed: 0 disconnect: 0 [ 2s ] executions total: 318 successful: 318 failed: 0 disconnect: 0 [ 3s ] executions total: 383 successful: 383 failed: 0 disconnect: 0 [ 4s ] executions total: 428 successful: 428 failed: 0 disconnect: 0 [ 5s ] executions total: 484 successful: 484 failed: 0 disconnect: 0 [ 6s ] executions total: 632 successful: 632 failed: 0 disconnect: 0 [ 7s ] executions total: 908 successful: 908 failed: 0 disconnect: 0 [ 8s ] executions total: 1092 successful: 1092 failed: 0 disconnect: 0 [ 9s ] executions total: 1341 successful: 1341 failed: 0 disconnect: 0 [ 10s ] executions total: 1552 successful: 1552 failed: 0 disconnect: 0 [ 11s ] executions total: 1764 successful: 1764 failed: 0 disconnect: 0 [ 12s ] executions total: 2007 successful: 2007 failed: 0 disconnect: 0 [ 13s ] executions total: 2198 successful: 2198 failed: 0 disconnect: 0 [ 14s ] executions total: 2230 successful: 2230 failed: 0 disconnect: 0 [ 15s ] executions total: 2287 successful: 2287 failed: 0 disconnect: 0 [ 16s ] executions total: 2322 successful: 2322 failed: 0 disconnect: 0 [ 17s ] executions total: 2359 successful: 2359 failed: 0 disconnect: 0 [ 18s ] executions total: 2393 successful: 2393 failed: 0 disconnect: 0 [ 19s ] executions total: 2429 successful: 2429 failed: 0 disconnect: 0 [ 20s ] executions total: 2467 successful: 2467 failed: 0 disconnect: 0 [ 21s ] executions total: 2497 successful: 2497 failed: 0 disconnect: 0 [ 22s ] executions total: 2539 successful: 2539 failed: 0 disconnect: 0 [ 23s ] executions total: 2594 successful: 2594 failed: 0 disconnect: 0 [ 24s ] executions total: 2691 successful: 2691 failed: 0 disconnect: 0 [ 25s ] executions total: 2906 successful: 2906 failed: 0 disconnect: 0 [ 26s ] executions total: 3156 successful: 3156 failed: 0 disconnect: 0 [ 27s ] executions total: 3392 successful: 3392 failed: 0 disconnect: 0 [ 28s ] executions total: 3627 successful: 3627 failed: 0 disconnect: 0 [ 29s ] executions total: 3870 successful: 3870 failed: 0 disconnect: 0 [ 30s ] executions total: 4110 successful: 4110 failed: 0 disconnect: 0 [ 31s ] executions total: 4314 successful: 4314 failed: 0 disconnect: 0 [ 32s ] executions total: 4505 successful: 4505 failed: 0 disconnect: 0 [ 33s ] executions total: 4564 successful: 4564 failed: 0 disconnect: 0 [ 34s ] executions total: 4614 successful: 4614 failed: 0 disconnect: 0 [ 35s ] executions total: 4744 successful: 4744 failed: 0 disconnect: 0 [ 36s ] executions total: 5014 successful: 5014 failed: 0 disconnect: 0 [ 37s ] executions total: 5243 successful: 5243 failed: 0 disconnect: 0 [ 38s ] executions total: 5511 successful: 5511 failed: 0 disconnect: 0 [ 39s ] executions total: 5778 successful: 5778 failed: 0 disconnect: 0 [ 40s ] executions total: 6004 successful: 6004 failed: 0 disconnect: 0 [ 41s ] executions total: 6280 successful: 6280 failed: 0 disconnect: 0 [ 42s ] executions total: 6519 successful: 6519 failed: 0 disconnect: 0 [ 43s ] executions total: 6737 successful: 6737 failed: 0 disconnect: 0 [ 44s ] executions total: 7001 successful: 7001 failed: 0 disconnect: 0 [ 45s ] executions total: 7261 successful: 7261 failed: 0 disconnect: 0 [ 46s ] executions total: 7518 successful: 7518 failed: 0 disconnect: 0 [ 47s ] executions total: 7759 successful: 7759 failed: 0 disconnect: 0 [ 48s ] executions total: 8006 successful: 8006 failed: 0 disconnect: 0 [ 49s ] executions total: 8248 successful: 8248 failed: 0 disconnect: 0 [ 50s ] executions total: 8462 successful: 8462 failed: 0 disconnect: 0 [ 51s ] executions total: 8735 successful: 8735 failed: 0 disconnect: 0 [ 52s ] executions total: 9000 successful: 9000 failed: 0 disconnect: 0 [ 53s ] executions total: 9242 successful: 9242 failed: 0 disconnect: 0 [ 54s ] executions total: 9491 successful: 9491 failed: 0 disconnect: 0 [ 55s ] executions total: 9749 successful: 9749 failed: 0 disconnect: 0 [ 56s ] executions total: 9962 successful: 9962 failed: 0 disconnect: 0 [ 57s ] executions total: 10228 successful: 10228 failed: 0 disconnect: 0 [ 58s ] executions total: 10443 successful: 10443 failed: 0 disconnect: 0 [ 59s ] executions total: 10674 successful: 10674 failed: 0 disconnect: 0 [ 60s ] executions total: 10828 successful: 10828 failed: 0 disconnect: 0 Test Result: Total Executions: 10828 Successful Executions: 10828 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: kingbase Host: kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local Port: 54321 Database: Table: User: esrep Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 10828 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-kingbase-jzvvix --namespace ns-svryh ` pod/test-db-client-executionloop-kingbase-jzvvix patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-kingbase-jzvvix" force deleted cluster update terminationPolicy WipeOut `kbcli cluster update kingbase-jzvvix --termination-policy=WipeOut --namespace ns-svryh ` cluster.apps.kubeblocks.io/kingbase-jzvvix updated check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:17 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0;standby: kingbase-jzvvix-kingbase-1 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done cluster full backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists get kingbase backup policy get backup policy:kingbase-jzvvix-kingbase-backup-policy success `kbcli cluster backup kingbase-jzvvix --method full --policy kingbase-jzvvix-kingbase-backup-policy --namespace ns-svryh ` Backup backup-ns-svryh-kingbase-jzvvix-20250619182656 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-svryh-kingbase-jzvvix-20250619182656 -n ns-svryh check backup status `kbcli cluster list-backups kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-svryh-kingbase-jzvvix-20250619182656 ns-svryh kingbase-jzvvix full Running Jun 19,2025 18:26 UTC+0800 backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running backup_status:kingbase-jzvvix-full-Running check backup status done backup_status:backup-ns-svryh-kingbase-jzvvix-20250619182656 ns-svryh kingbase-jzvvix full Completed 13939026 83s Jun 19,2025 18:26 UTC+0800 Jun 19,2025 18:28 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "kingbase-jzvvix-backup" not found `kbcli cluster describe-backup backup-ns-svryh-kingbase-jzvvix-20250619182656 --namespace ns-svryh ` Name: backup-ns-svryh-kingbase-jzvvix-20250619182656 Cluster: kingbase-jzvvix Namespace: ns-svryh Spec: Method: full Policy Name: kingbase-jzvvix-kingbase-backup-policy Status: Phase: Completed Total Size: 13939026 ActionSet Name: kingbase-full-backup Repository: backuprepo-kbcli-test Duration: 83s Start Time: Jun 19,2025 18:26 UTC+0800 Completion Time: Jun 19,2025 18:28 UTC+0800 Path: /ns-svryh/kingbase-jzvvix-c138256e-6407-4888-8eef-428d38fc1672/kingbase/backup-ns-svryh-kingbase-jzvvix-20250619182656 Time Range Start: Jun 19,2025 08:00 UTC+0800 Time Range End: Jun 19,2025 08:00 UTC+0800 Warning Events: `kbcli cluster restore kingbase-jzvvix-backup --backup backup-ns-svryh-kingbase-jzvvix-20250619182656 --namespace ns-svryh ` Cluster kingbase-jzvvix-backup created check cluster status `kbcli cluster list kingbase-jzvvix-backup --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix-backup ns-svryh kingbase WipeOut Creating Jun 19,2025 18:28 UTC+0800 clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix-backup --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-backup-kingbase-0 ns-svryh kingbase-jzvvix-backup kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-11-25.us-west-2.compute.internal/172.31.11.25 Jun 19,2025 18:29 UTC+0800 etc:1Gi kingbase-jzvvix-backup-kingbase-1 ns-svryh kingbase-jzvvix-backup kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:30 UTC+0800 etc:1Gi kingbase-jzvvix-backup-kingbase-2 ns-svryh kingbase-jzvvix-backup kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 18:32 UTC+0800 etc:1Gi kingbase-jzvvix-backup-kmonitor-0 ns-svryh kingbase-jzvvix-backup kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 18:28 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-backup-kingbase-0;standby: kingbase-jzvvix-backup-kingbase-1 kingbase-jzvvix-backup-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix-backup` set secret: kingbase-jzvvix-backup-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-backup-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-backup-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-backup-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-backup-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kbcli cluster describe-backup backup-ns-svryh-kingbase-jzvvix-20250619182656 --namespace ns-svryh ` Name: backup-ns-svryh-kingbase-jzvvix-20250619182656 Cluster: kingbase-jzvvix Namespace: ns-svryh Spec: Method: full Policy Name: kingbase-jzvvix-kingbase-backup-policy Status: Phase: Completed Total Size: 13939026 ActionSet Name: kingbase-full-backup Repository: backuprepo-kbcli-test Duration: 83s Start Time: Jun 19,2025 18:26 UTC+0800 Completion Time: Jun 19,2025 18:28 UTC+0800 Path: /ns-svryh/kingbase-jzvvix-c138256e-6407-4888-8eef-428d38fc1672/kingbase/backup-ns-svryh-kingbase-jzvvix-20250619182656 Time Range Start: Jun 19,2025 08:00 UTC+0800 Time Range End: Jun 19,2025 08:00 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix-backup` set secret: kingbase-jzvvix-backup-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-backup-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-backup-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-backup-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. `echo "echo \"show max_connections;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-backup-kingbase-0 --namespace ns-svryh -- bash ` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file max_connections ----------------- 100 (1 row) connect cluster Success set max_connections to 100 delete cluster kingbase-jzvvix-backup `kbcli cluster delete kingbase-jzvvix-backup --auto-approve --namespace ns-svryh ` Cluster kingbase-jzvvix-backup deleted Error from server (NotFound): secrets "kingbase-jzvvix-backup-license" not found Error from server (NotFound): secrets "kingbase-jzvvix-backup-license" not found Error from server (NotFound): secrets "kingbase-jzvvix-backup-license" not found pod_info:kingbase-jzvvix-backup-kingbase-0 2/3 Terminating 1 (2m42s ago) 7m36s kingbase-jzvvix-backup-kingbase-1 2/3 Terminating 0 6m31s kingbase-jzvvix-backup-kingbase-2 2/3 Terminating 0 5m16s kingbase-jzvvix-backup-kmonitor-0 1/1 Terminating 0 8m50s pod_info:kingbase-jzvvix-backup-kingbase-0 2/3 Terminating 1 (3m3s ago) 7m57s kingbase-jzvvix-backup-kingbase-1 2/3 Terminating 0 6m52s kingbase-jzvvix-backup-kingbase-2 2/3 Terminating 0 5m37s kingbase-jzvvix-backup-kmonitor-0 1/1 Terminating 0 9m11s No resources found in ns-svryh namespace. delete cluster pod done No resources found in ns-svryh namespace. check cluster resource non-exist OK: pvc No resources found in ns-svryh namespace. delete cluster done No resources found in ns-svryh namespace. No resources found in ns-svryh namespace. No resources found in ns-svryh namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-svryh-kingbase-jzvvix-20250619182656 --namespace ns-svryh ` backup.dataprotection.kubeblocks.io/backup-ns-svryh-kingbase-jzvvix-20250619182656 patched `kbcli cluster delete-backup kingbase-jzvvix --name backup-ns-svryh-kingbase-jzvvix-20250619182656 --force --auto-approve --namespace ns-svryh ` Backup backup-ns-svryh-kingbase-jzvvix-20250619182656 deleted No opsrequests found in ns-svryh namespace. test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote kingbase-jzvvix --auto-approve --force=true --component kingbase --instance kingbase-jzvvix-kingbase-2 --namespace ns-svryh ` component:kingbase OpsRequest kingbase-jzvvix-switchover-6jhrr created successfully, you can view the progress: kbcli cluster describe-ops kingbase-jzvvix-switchover-6jhrr -n ns-svryh check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 ops_status:kingbase-jzvvix-switchover-6jhrr ns-svryh Switchover kingbase-jzvvix Pending -/- Jun 19,2025 18:38 UTC+0800 check ops status timeout --------------------------------------get cluster kingbase-jzvvix yaml-------------------------------------- `kubectl get cluster kingbase-jzvvix -o yaml --namespace ns-svryh ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"kingbase-jzvvix-switchover-6jhrr","type":"Switchover"***]' kubeblocks.io/reconcile: "2025-06-19T10:21:07.551177998Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"kingbase-jzvvix","namespace":"ns-svryh"***,"spec":***"clusterDefinitionRef":"kingbase","componentSpecs":[***"name":"kingbase","replicas":3,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"schedulingPolicy":***"affinity":***"podAntiAffinity":***"requiredDuringSchedulingIgnoredDuringExecution":[***"labelSelector":***"matchLabels":***"app.kubernetes.io/instance":"kingbase-jzvvix","apps.kubeblocks.io/component-name":"kingbase"***,"topologyKey":"kubernetes.io/hostname"***]***,"serviceVersion":"8.6.8","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"1Gi"***,"storageClassName":null***,***"name":"etc","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"1Gi"***,"storageClassName":null***],"volumes":[***"name":"license","secret":***"secretName":"kingbase-jzvvix-license"***]***,***"name":"kmonitor","replicas":1,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"schedulingPolicy":***"affinity":***"podAntiAffinity":***"requiredDuringSchedulingIgnoredDuringExecution":[***"labelSelector":***"matchLabels":***"app.kubernetes.io/instance":"kingbase-jzvvix","apps.kubeblocks.io/component-name":"kmonitor"***,"topologyKey":"kubernetes.io/hostname"***]***,"serviceVersion":"8.6.8","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"1Gi"***,"storageClassName":null***,***"name":"etc","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"1Gi"***,"storageClassName":null***]***],"terminationPolicy":"DoNotTerminate","topology":"kingbase-cluster"*** creationTimestamp: "2025-06-19T10:16:44Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 4 labels: app.kubernetes.io/instance: kingbase-jzvvix clusterdefinition.kubeblocks.io/name: kingbase clusterversion.kubeblocks.io/name: "" name: kingbase-jzvvix namespace: ns-svryh resourceVersion: "44216" uid: c138256e-6407-4888-8eef-428d38fc1672 spec: clusterDefinitionRef: kingbase componentSpecs: - componentDef: kingbase name: kingbase replicas: 3 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi schedulingPolicy: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app.kubernetes.io/instance: kingbase-jzvvix apps.kubeblocks.io/component-name: kingbase topologyKey: kubernetes.io/hostname serviceVersion: 8.6.8 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: etc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi volumes: - name: license secret: secretName: kingbase-jzvvix-license - componentDef: kingbase-monitor name: kmonitor replicas: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi schedulingPolicy: affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchLabels: app.kubernetes.io/instance: kingbase-jzvvix apps.kubeblocks.io/component-name: kmonitor topologyKey: kubernetes.io/hostname serviceVersion: 8.6.8 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: etc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: WipeOut topology: kingbase-cluster status: clusterDefGeneration: 1 components: kingbase: phase: Running podsReady: true podsReadyTime: "2025-06-19T10:33:36Z" kmonitor: phase: Running podsReady: true podsReadyTime: "2025-06-19T10:17:39Z" conditions: - lastTransitionTime: "2025-06-19T10:16:44Z" message: 'The operator has started the provisioning of Cluster: kingbase-jzvvix' observedGeneration: 4 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:16:44Z" message: Successfully applied for resources observedGeneration: 4 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T10:33:36Z" message: all pods of components are ready, waiting for the probe detection successful reason: AllReplicasReady status: "True" type: ReplicasReady - lastTransitionTime: "2025-06-19T10:33:36Z" message: 'Cluster: kingbase-jzvvix is ready, current phase is Running' reason: ClusterReady status: "True" type: Ready observedGeneration: 4 phase: Running ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster kingbase-jzvvix-------------------------------------- `kubectl describe cluster kingbase-jzvvix --namespace ns-svryh ` Name: kingbase-jzvvix Namespace: ns-svryh Labels: app.kubernetes.io/instance=kingbase-jzvvix clusterdefinition.kubeblocks.io/name=kingbase clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"kingbase-jzvvix-switchover-6jhrr","type":"Switchover"***] kubeblocks.io/reconcile: 2025-06-19T10:21:07.551177998Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:16:44Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 4 Resource Version: 44216 UID: c138256e-6407-4888-8eef-428d38fc1672 Spec: Cluster Definition Ref: kingbase Component Specs: Component Def: kingbase Name: kingbase Replicas: 3 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Scheduling Policy: Affinity: Pod Anti Affinity: Required During Scheduling Ignored During Execution: Label Selector: Match Labels: app.kubernetes.io/instance: kingbase-jzvvix apps.kubeblocks.io/component-name: kingbase Topology Key: kubernetes.io/hostname Service Version: 8.6.8 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 1Gi Name: etc Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 1Gi Volumes: Name: license Secret: Secret Name: kingbase-jzvvix-license Component Def: kingbase-monitor Name: kmonitor Replicas: 1 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Scheduling Policy: Affinity: Pod Anti Affinity: Required During Scheduling Ignored During Execution: Label Selector: Match Labels: app.kubernetes.io/instance: kingbase-jzvvix apps.kubeblocks.io/component-name: kmonitor Topology Key: kubernetes.io/hostname Service Version: 8.6.8 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 1Gi Name: etc Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 1Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: WipeOut Topology: kingbase-cluster Status: Cluster Def Generation: 1 Components: Kingbase: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T10:33:36Z Kmonitor: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T10:17:39Z Conditions: Last Transition Time: 2025-06-19T10:16:44Z Message: The operator has started the provisioning of Cluster: kingbase-jzvvix Observed Generation: 4 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:16:44Z Message: Successfully applied for resources Observed Generation: 4 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T10:33:36Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2025-06-19T10:33:36Z Message: Cluster: kingbase-jzvvix is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 4 Phase: Running Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 29m (x2 over 29m) cluster-controller component is Creating Warning Unhealthy 25m event-controller Pod kingbase-jzvvix-kingbase-0: Liveness probe failed: dial tcp 172.31.5.253:54321: connect: connection refused Warning ReplicasNotReady 14m (x2 over 29m) cluster-controller pods are not ready in Components: [kingbase], refer to related component message in Cluster.status.components Warning ComponentsNotReady 14m (x2 over 29m) cluster-controller pods are unavailable in Components: [kingbase], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 14m cluster-controller component is Updating Normal ComponentPhaseTransition 13m (x3 over 29m) cluster-controller component is Running Normal AllReplicasReady 13m (x2 over 23m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 13m (x2 over 23m) cluster-controller Cluster: kingbase-jzvvix is ready, current phase is Running Normal Running 13m (x2 over 23m) cluster-controller Cluster: kingbase-jzvvix is ready, current phase is Running Normal PreCheckSucceed 8m23s (x4 over 30m) cluster-controller The operator has started the provisioning of Cluster: kingbase-jzvvix Normal ApplyResourcesSucceed 8m23s (x4 over 30m) cluster-controller Successfully applied for resources ------------------------------------------------------------------------------------------------------------------ `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests kingbase-jzvvix-switchover-6jhrr --namespace ns-svryh ` opsrequest.apps.kubeblocks.io/kingbase-jzvvix-switchover-6jhrr patched `kbcli cluster delete-ops --name kingbase-jzvvix-switchover-6jhrr --force --auto-approve --namespace ns-svryh ` OpsRequest kingbase-jzvvix-switchover-6jhrr deleted check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` No opsrequests found in ns-svryh namespace. No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: No opsrequests found in ns-svryh namespace. ops_status: not found ops Switchover|Custom switchover pod:kingbase-jzvvix-kingbase-0 checking switchover... check switchover timeout cluster update monitor KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster kingbase-jzvvix --namespace ns-svryh --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/kingbase-jzvvix patched check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done create curl-jq pod kingbase-jzvvix-kb-curl `kubectl run --namespace kb-mqtcr kingbase-jzvvix-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/kingbase-jzvvix-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE kingbase-jzvvix-kb-curl 0/1 ContainerCreating 0 1s check pod kingbase-jzvvix-kb-curl status done `kubectl wait --for=condition=Ready pod kingbase-jzvvix-kb-curl --namespace kb-mqtcr` pod/kingbase-jzvvix-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE kingbase-jzvvix-kb-curl 1/1 Running 0 6s kubectl exec -it kingbase-jzvvix-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it kingbase-jzvvix-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 19110 0 --:--:-- --:--:-- --:--:-- 20333 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750330302.854,"24.700000000000003"]***]*** k8s_pod_cpu_limit: 1750330302.854 monitor cluster Success `kubectl delete pod kingbase-jzvvix-kb-curl --namespace kb-mqtcr --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "kingbase-jzvvix-kb-curl" force deleted check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kingbase-jzvvix" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-kingbase-jzvvix.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-kingbase-jzvvix created apply test-chaos-mesh-networklossover-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-networklossover-kingbase-jzvvix.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kingbase-jzvvix" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-kingbase-jzvvix" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-kingbase-jzvvix --namespace ns-svryh ` Error from server (NotFound): pods "test-db-client-connectionstress-kingbase-jzvvix" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-kingbase-jzvvix" not found `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-kingbase-jzvvix namespace: ns-svryh spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local" - "--user" - "esrep" - "--password" - "Kingbaseha110" - "--port" - "54321" - "--database" - "esrep" - "--dbtype" - "kingbase" - "--test" - "connectionstress" - "--connections" - "100" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-kingbase-jzvvix.yaml` pod/test-db-client-connectionstress-kingbase-jzvvix created apply test-db-client-connectionstress-kingbase-jzvvix.yaml Success `rm -rf test-db-client-connectionstress-kingbase-jzvvix.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kingbase-jzvvix 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kingbase-jzvvix 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kingbase-jzvvix 1/1 Running 0 17s check pod test-db-client-connectionstress-kingbase-jzvvix status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kingbase-jzvvix 0/1 Completed 0 24s check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done --host kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local --user esrep --password Kingbaseha110 --port 54321 --database esrep --dbtype kingbase --test connectionstress --connections 100 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:57:07.056 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Test Result: null Connection Information: Database Type: kingbase Host: kingbase-jzvvix-kingbase-kingbase-es.ns-svryh.svc.cluster.local Port: 54321 Database: esrep Table: User: esrep Org: Access Mode: mysql Test Type: connectionstress Connection Count: 100 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-kingbase-jzvvix --namespace ns-svryh ` pod/test-db-client-connectionstress-kingbase-jzvvix patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-kingbase-jzvvix" force deleted check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kingbase-jzvvix" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-kingbase-jzvvix.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-kingbase-jzvvix created apply test-chaos-mesh-podfailure-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-podfailure-kingbase-jzvvix.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Abnormal Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kingbase-jzvvix" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kingbase-jzvvix" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-kingbase-jzvvix.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-kingbase-jzvvix created apply test-chaos-mesh-timeoffset-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-timeoffset-kingbase-jzvvix.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kingbase-jzvvix" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kingbase-jzvvix" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-kingbase-jzvvix.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-kingbase-jzvvix created apply test-chaos-mesh-networkcorruptover-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-kingbase-jzvvix.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kingbase-jzvvix" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kingbase-jzvvix" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-kingbase-jzvvix.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-kingbase-jzvvix created apply test-chaos-mesh-dnserror-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-dnserror-kingbase-jzvvix.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Abnormal Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kingbase-jzvvix" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kingbase-jzvvix" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-kingbase-jzvvix.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-kingbase-jzvvix created apply test-chaos-mesh-networkdelay-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-networkdelay-kingbase-jzvvix.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 18:31 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kingbase-jzvvix" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover delete pod:kingbase-jzvvix-kingbase-0 `kubectl delete pod kingbase-jzvvix-kingbase-0 --namespace ns-svryh ` pod "kingbase-jzvvix-kingbase-0" deleted check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:14 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 18:19 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale kingbase-jzvvix --auto-approve --force=true --components kingbase --cpu 600m --memory 0.6Gi --namespace ns-svryh ` OpsRequest kingbase-jzvvix-verticalscaling-jv46h created successfully, you can view the progress: kbcli cluster describe-ops kingbase-jzvvix-verticalscaling-jv46h -n ns-svryh check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-verticalscaling-jv46h ns-svryh VerticalScaling kingbase-jzvvix kingbase Running 0/3 Jun 19,2025 19:16 UTC+0800 check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:19 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:18 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:16 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-144.us-west-2.compute.internal/172.31.5.144 Jun 19,2025 18:16 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-verticalscaling-jv46h ns-svryh VerticalScaling kingbase-jzvvix kingbase Succeed 3/3 Jun 19,2025 19:16 UTC+0800 check ops status done ops_status:kingbase-jzvvix-verticalscaling-jv46h ns-svryh VerticalScaling kingbase-jzvvix kingbase Succeed 3/3 Jun 19,2025 19:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests kingbase-jzvvix-verticalscaling-jv46h --namespace ns-svryh ` opsrequest.apps.kubeblocks.io/kingbase-jzvvix-verticalscaling-jv46h patched `kbcli cluster delete-ops --name kingbase-jzvvix-verticalscaling-jv46h --force --auto-approve --namespace ns-svryh ` OpsRequest kingbase-jzvvix-verticalscaling-jv46h deleted `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success 11 cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart kingbase-jzvvix --auto-approve --force=true --namespace ns-svryh ` OpsRequest kingbase-jzvvix-restart-cpd2t created successfully, you can view the progress: kbcli cluster describe-ops kingbase-jzvvix-restart-cpd2t -n ns-svryh check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-restart-cpd2t ns-svryh Restart kingbase-jzvvix kingbase,kmonitor Running 0/4 Jun 19,2025 19:20 UTC+0800 check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-restart-cpd2t ns-svryh Restart kingbase-jzvvix kingbase,kmonitor Succeed 4/4 Jun 19,2025 19:20 UTC+0800 check ops status done ops_status:kingbase-jzvvix-restart-cpd2t ns-svryh Restart kingbase-jzvvix kingbase,kmonitor Succeed 4/4 Jun 19,2025 19:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests kingbase-jzvvix-restart-cpd2t --namespace ns-svryh ` opsrequest.apps.kubeblocks.io/kingbase-jzvvix-restart-cpd2t patched `kbcli cluster delete-ops --name kingbase-jzvvix-restart-cpd2t --force --auto-approve --namespace ns-svryh ` OpsRequest kingbase-jzvvix-restart-cpd2t deleted `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done check failover pod name failover pod name:kingbase-jzvvix-kingbase-1 checking failover... `kill 1` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done failover pod name:kingbase-jzvvix-kingbase-1 checking failover... `kill 1` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done failover pod name:kingbase-jzvvix-kingbase-1 checking failover... `kill 1` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done failover pod name:kingbase-jzvvix-kingbase-1 checking failover... `kill 1` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done failover pod name:kingbase-jzvvix-kingbase-1 checking failover... `kill 1` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done check failover pod name timeout test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kingbase-jzvvix" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-kingbase-jzvvix.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-kingbase-jzvvix created apply test-chaos-mesh-fullcpu-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-fullcpu-kingbase-jzvvix.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kingbase-jzvvix" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-1 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kingbase-jzvvix" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-kingbase-jzvvix.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-kingbase-jzvvix created apply test-chaos-mesh-networkbandwidthover-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-kingbase-jzvvix.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kingbase-jzvvix" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kingbase-jzvvix" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 action: partition mode: all target: mode: all selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-2 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-kingbase-jzvvix.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-kingbase-jzvvix created apply test-chaos-mesh-networkpartition-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-networkpartition-kingbase-jzvvix.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:1Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kingbase-jzvvix" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=kingbase-jzvvix,apps.kubeblocks.io/component-name=kingbase,apps.kubeblocks.io/vct-name=data --namespace ns-svryh ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in kingbase-jzvvix namespace. `kbcli cluster volume-expand kingbase-jzvvix --auto-approve --force=true --components kingbase --volume-claim-templates data --storage 3Gi --namespace ns-svryh ` OpsRequest kingbase-jzvvix-volumeexpansion-df4sw created successfully, you can view the progress: kbcli cluster describe-ops kingbase-jzvvix-volumeexpansion-df4sw -n ns-svryh check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-volumeexpansion-df4sw ns-svryh VolumeExpansion kingbase-jzvvix kingbase Running 0/3 Jun 19,2025 19:37 UTC+0800 check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done No resources found in kingbase-jzvvix namespace. check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-volumeexpansion-df4sw ns-svryh VolumeExpansion kingbase-jzvvix kingbase Succeed 3/3 Jun 19,2025 19:37 UTC+0800 check ops status done ops_status:kingbase-jzvvix-volumeexpansion-df4sw ns-svryh VolumeExpansion kingbase-jzvvix kingbase Succeed 3/3 Jun 19,2025 19:37 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests kingbase-jzvvix-volumeexpansion-df4sw --namespace ns-svryh ` opsrequest.apps.kubeblocks.io/kingbase-jzvvix-volumeexpansion-df4sw patched `kbcli cluster delete-ops --name kingbase-jzvvix-volumeexpansion-df4sw --force --auto-approve --namespace ns-svryh ` OpsRequest kingbase-jzvvix-volumeexpansion-df4sw deleted `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kingbase-jzvvix" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-kingbase-jzvvix.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-kingbase-jzvvix created apply test-chaos-mesh-dnsrandom-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-dnsrandom-kingbase-jzvvix.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:24 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kingbase-jzvvix" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod kingbase-jzvvix-kingbase-0 --namespace ns-svryh -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-10-248.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod kingbase-jzvvix-kingbase-1 --namespace ns-svryh -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-2-66.us-west-2.compute.internal success kubectl get pod kingbase-jzvvix-kingbase-2 --namespace ns-svryh -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-6-5.us-west-2.compute.internal success kubectl drain ip-172-31-10-248.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-10-248.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-vs2tk, kb-mqtcr/kb-addon-apecloud-otel-collector-t9td4, kube-system/aws-node-qghnb, kube-system/ebs-csi-node-hfpjq, kube-system/kube-proxy-l2krz evicting pod ns-svryh/kingbase-jzvvix-kingbase-0 evicting pod kb-mqtcr/kubeblocks-58f8dff998-dqfwk pod/kingbase-jzvvix-kingbase-0 evicted pod/kubeblocks-58f8dff998-dqfwk evicted node/ip-172-31-10-248.us-west-2.compute.internal drained kubectl uncordon ip-172-31-10-248.us-west-2.compute.internal node/ip-172-31-10-248.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 19:42 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0 kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kingbase-jzvvix" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-kingbase-jzvvix.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-kingbase-jzvvix created apply test-chaos-mesh-oom-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-oom-kingbase-jzvvix.yaml` check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 19:42 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:23 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:21 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-1;standby: kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kingbase-jzvvix" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-2 --namespace ns-svryh -- bash` check readonly db_client batch data Success cluster configure component_tmp: kingbase check cluster status before ops cluster_status:Updating check cluster status done cluster_status:Running `kbcli cluster configure kingbase-jzvvix --auto-approve --force=true --set max_connections=200 --components kingbase --config-spec kingbase-config --config-file kingbase.conf --namespace ns-svryh ` Will updated configure file meta: ConfigSpec: kingbase-config ConfigFile: kingbase.conf ComponentName: kingbase ClusterName: kingbase-jzvvix OpsRequest kingbase-jzvvix-reconfiguring-69z95 created successfully, you can view the progress: kbcli cluster describe-ops kingbase-jzvvix-reconfiguring-69z95 -n ns-svryh check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-reconfiguring-69z95 ns-svryh Reconfiguring kingbase-jzvvix kingbase Running -/- Jun 19,2025 19:45 UTC+0800 check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:49 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:47 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:46 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0;standby: kingbase-jzvvix-kingbase-1 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done check ops status `kbcli cluster list-ops kingbase-jzvvix --status all --namespace ns-svryh ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kingbase-jzvvix-reconfiguring-69z95 ns-svryh Reconfiguring kingbase-jzvvix kingbase Succeed 2/2 Jun 19,2025 19:45 UTC+0800 check ops status done ops_status:kingbase-jzvvix-reconfiguring-69z95 ns-svryh Reconfiguring kingbase-jzvvix kingbase Succeed 2/2 Jun 19,2025 19:45 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests kingbase-jzvvix-reconfiguring-69z95 --namespace ns-svryh ` opsrequest.apps.kubeblocks.io/kingbase-jzvvix-reconfiguring-69z95 patched `kbcli cluster delete-ops --name kingbase-jzvvix-reconfiguring-69z95 --force --auto-approve --namespace ns-svryh ` OpsRequest kingbase-jzvvix-reconfiguring-69z95 deleted check config variables `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 200 configure:[max_connections] result actual:[200] equal expected:[200] `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kingbase-jzvvix" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kingbase-jzvvix" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-kingbase-jzvvix namespace: ns-svryh spec: selector: namespaces: - ns-svryh labelSelectors: apps.kubeblocks.io/pod-name: kingbase-jzvvix-kingbase-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-kingbase-jzvvix.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-kingbase-jzvvix created apply test-chaos-mesh-networkduplicate-kingbase-jzvvix.yaml Success `rm -rf test-chaos-mesh-networkduplicate-kingbase-jzvvix.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list kingbase-jzvvix --show-labels --namespace ns-svryh ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS kingbase-jzvvix ns-svryh kingbase WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=kingbase-jzvvix,clusterdefinition.kubeblocks.io/name=kingbase,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kingbase-jzvvix --namespace ns-svryh ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kingbase-jzvvix-kingbase-0 ns-svryh kingbase-jzvvix kingbase Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 19:49 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-1 ns-svryh kingbase-jzvvix kingbase Running standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-10-248.us-west-2.compute.internal/172.31.10.248 Jun 19,2025 19:47 UTC+0800 etc:1Gi kingbase-jzvvix-kingbase-2 ns-svryh kingbase-jzvvix kingbase Terminating standby us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-5.us-west-2.compute.internal/172.31.6.5 Jun 19,2025 19:46 UTC+0800 etc:1Gi kingbase-jzvvix-kmonitor-0 ns-svryh kingbase-jzvvix kmonitor Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-15.us-west-2.compute.internal/172.31.2.15 Jun 19,2025 19:21 UTC+0800 etc:1Gi pod_status:Terminating pod_status:Terminating pod_status:Terminating pod_status:Terminating pod_status:Terminating pod_status:Init:0/3 pod_status:Init:0/3 pod_status:Init:0/3 pod_status:PodInitializing pod_status:PodInitializing pod_status:PodInitializing pod_status:PodInitializing pod_status:PodInitializing check pod status done check cluster role check cluster role done primary: kingbase-jzvvix-kingbase-0;standby: kingbase-jzvvix-kingbase-1 kingbase-jzvvix-kingbase-2 `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check cluster connect `echo " /var/lib/data/bin/ksql -U esrep -p 54321 -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-kingbase-jzvvix --namespace ns-svryh ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kingbase-jzvvix" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kingbase-jzvvix" not found check failover pod name failover pod name:kingbase-jzvvix-kingbase-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep No resources found in ns-svryh namespace. check db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-0 --namespace ns-svryh -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=kingbase-jzvvix` set secret: kingbase-jzvvix-kingbase-account-esrep `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.username***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.password***"` `kubectl get secrets kingbase-jzvvix-kingbase-account-esrep -o jsonpath="***.data.port***"` DB_USERNAME:esrep;DB_PASSWORD:Kingbaseha110;DB_PORT:54321;DB_DATABASE:esrep check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop_table;\" | /var/lib/data/bin/ksql -U esrep -p 54321 -d executions_loop -w " | kubectl exec -it kingbase-jzvvix-kingbase-1 --namespace ns-svryh -- bash` check readonly db_client batch data Success cluster list-logs `kbcli cluster list-logs kingbase-jzvvix --namespace ns-svryh ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update kingbase-jzvvix --enable-all-logs=true --namespace ns-svryh Error from server (NotFound): pods "kingbase-jzvvix-kingbase-0" not found cluster logs `kbcli cluster logs kingbase-jzvvix --tail 30 --namespace ns-svryh ` Defaulted container "kingbase" out of: kingbase, lorry, config-manager, init (init), init-lorry (init), install-config-manager-tool (init) [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Permit User Sessions... Starting Login Service... Starting OpenSSH Server Key Generation... [ OK ] Started D-Bus System Message Bus. Starting Dump dmesg to /var/log/dmesg... Starting LSB: Bring up/down networking... [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Permit User Sessions. Starting Cleanup of Temporary Directories... [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Dump dmesg to /var/log/dmesg. [FAILED] Failed to start LSB: Bring up/down networking. See 'systemctl status network.service' for details. [ OK ] Started Cleanup of Temporary Directories. [ OK ] Started Login Service. [ OK ] Reached target Network. [ OK ] Reached target Network is Online. [ OK ] Started OpenSSH Server Key Generation. Starting OpenSSH server daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Reached target Multi-User System. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. delete cluster kingbase-jzvvix `kbcli cluster delete kingbase-jzvvix --auto-approve --namespace ns-svryh ` Cluster kingbase-jzvvix deleted secret "kingbase-jzvvix-license" deleted pod_info:kingbase-jzvvix-kingbase-0 3/3 Running 0 6m15s kingbase-jzvvix-kingbase-1 3/3 Running 0 7m43s kingbase-jzvvix-kingbase-2 3/3 Running 0 105s kingbase-jzvvix-kmonitor-0 1/1 Terminating 0 33m pod_info:kingbase-jzvvix-kingbase-0 2/3 Terminating 0 6m36s kingbase-jzvvix-kingbase-1 2/3 Terminating 0 8m4s kingbase-jzvvix-kingbase-2 2/3 Terminating 0 2m6s kingbase-jzvvix-kmonitor-0 1/1 Terminating 0 34m No resources found in ns-svryh namespace. delete cluster pod done No resources found in ns-svryh namespace. check cluster resource non-exist OK: pvc No resources found in ns-svryh namespace. delete cluster done No resources found in ns-svryh namespace. No resources found in ns-svryh namespace. No resources found in ns-svryh namespace. Kingbase Test Suite All Done! --------------------------------------Kingbase (Topology = kingbase-cluster Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=kingbase-cluster;ComponentVersion=kingbase-cluster;ServiceVersion=8.6.8;]|[Description=Create a cluster with the specified topology kingbase-cluster and component version kingbase-cluster and service version 8.6.8] [PASSED]|[Connect]|[ComponentName=kingbase]|[Description=Connect to the cluster] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=full]|[Description=The cluster full Backup] [PASSED]|[Restore]|[BackupMethod=full]|[Description=The cluster full Restore] [PASSED]|[Connect]|[ComponentName=kingbase]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=full]|[Description=Delete the full restore cluster] [FAILED]|[SwitchOver]|[ComponentName=kingbase]|[Description=SwitchOver the cluster specify component kingbase] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=kingbase]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=kingbase]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=kingbase]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=kingbase]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=kingbase]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=kingbase]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=kingbase]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=kingbase]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[VerticalScaling]|[ComponentName=kingbase]|[Description=VerticalScaling the cluster specify component kingbase] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [FAILED]|[Failover]|[HA=Kill 1;ComponentName=kingbase]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=kingbase]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=kingbase]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=kingbase]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[VolumeExpansion]|[ComponentName=kingbase]|[Description=VolumeExpansion the cluster specify component kingbase] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=kingbase]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=kingbase]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=kingbase]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Reconfiguring]|[ComponentName=kingbase;max_connections=200]|[Description=Reconfiguring the cluster specify component kingbase set max_connections=200] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=kingbase]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]