source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-gdyfe ` `kubectl create namespace ns-gdyfe` namespace/ns-gdyfe created create namespace ns-gdyfe done check registry secret No resources found in ns-gdyfe namespace. secret/kbcli-test-registry-key created create docker registry secret kbcli-test-registry-key Success check ns:ns-gdyfe sa:default imagePullSecrets patch ns:ns-gdyfe sa:default imagePullSecrets serviceaccount/default patched patch ns:ns-gdyfe sa:default imagePullSecrets Success download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 90.4M 0 --:--:-- --:--:-- --:--:-- 90.4M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition Error from server (NotFound): clusterdefinitions.apps.kubeblocks.io "goldendb" not found KUBEBLOCKS NAMESPACE:kb-devqu get kubeblocks namespace done annotate goldendb cmpd apps.kubeblocks.io/skip-immutable-check=true No resources found get helm repo list success `helm repo remove kbcli-test-enterprise ` Error: no repo named "kbcli-test-enterprise" found "kbcli-test-enterprise" has been added to your repositories add helm chart repo success `helm repo update kbcli-test-enterprise ` Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "kbcli-test-enterprise" chart repository Update Complete. ⎈Happy Helming!�� KUBEBLOCKS NAMESPACE:kb-devqu get kubeblocks namespace done `helm upgrade --install --namespace kb-devqu kb-addon-goldendb kbcli-test-enterprise/goldendb --version 0.9.0 --set image.registry=docker.io --set images.registry=docker.io ` Release "kb-addon-goldendb" does not exist. Installing it now. NAME: kb-addon-goldendb LAST DEPLOYED: Thu Jun 19 18:15:24 2025 NAMESPACE: kb-devqu STATUS: deployed REVISION: 1 NOTES: Release Information: Commit ID: "f272b65eab20aa65cd70c5ebf83fec8aff43e62d" Commit Time: "2025-06-13 18:33:47 +0800" Release Branch: "v0.9.4" Release Time: "2025-06-19 17:31:44 +0800" Enterprise: "true" set component version set component version:goldendb-cluster set service versions:6.1.3 set service versions sorted:6.1.3 KUBEBLOCKS NAMESPACE:kb-devqu get kubeblocks namespace done patch goldendb cmpd imagePullSecrets componentdefinition.apps.kubeblocks.io/golden-cn-6.1 patched componentdefinition.apps.kubeblocks.io/golden-dn-6.1 patched componentdefinition.apps.kubeblocks.io/golden-gtm-6.1 patched componentdefinition.apps.kubeblocks.io/golden-lds-6.1 patched componentdefinition.apps.kubeblocks.io/golden-manager-6.1 patched componentdefinition.apps.kubeblocks.io/golden-zk-6.1 patched check clusterdefinition goldendb status done set component name:goldendb-dn-00 found component version no cluster version found unsupported component definition set replicas first:2,6.1.3 set replicas third:2,6.1.3 set replicas fourth:2,6.1.3 set minimum cmpv service version set minimum cmpv service version replicas:2,6.1.3 REPORT_COUNT:1 CLUSTER_TOPOLOGY:goldendb-cluster topology goldendb-cluster found in cluster definition goldendb LIMIT_CPU:1 LIMIT_MEMORY:1 storage size: 10 No resources found in ns-gdyfe namespace. termination_policy:WipeOut create 2 replica WipeOut goldendb cluster check cluster definition apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: goldendb-wcalnk namespace: ns-gdyfe spec: clusterDefinitionRef: goldendb topology: goldendb-cluster terminationPolicy: WipeOut componentSpecs: - name: goldendb-dn-00 serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 annotations: "gdb.timezone": Asia/Shanghai env: - name: TZ value: Asia/Shanghai volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File replicas: 2 resources: limits: cpu: 1000m memory: 2Gi requests: cpu: 1000m memory: 2Gi volumeClaimTemplates: - name: backup spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-agent spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-log-mysql-log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-mysql-data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: goldendb-cn-00 serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 annotations: "gdb.timezone": Asia/Shanghai env: - name: TZ value: Asia/Shanghai volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File replicas: 2 resources: limits: cpu: 1000m memory: 3Gi requests: cpu: 1000m memory: 3Gi volumeClaimTemplates: - name: data-agent spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-cn spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: goldendb-zk serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 annotations: "gdb.timezone": Asia/Shanghai env: - name: TZ value: Asia/Shanghai volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File replicas: 1 resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 1000m memory: 1Gi volumeClaimTemplates: - name: insight-data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: goldendb-manager serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 annotations: "gdb.timezone": Asia/Shanghai env: - name: TZ value: Asia/Shanghai volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File replicas: 1 resources: limits: cpu: 3000m memory: 8Gi requests: cpu: 3000m memory: 8Gi volumeClaimTemplates: - name: insight-data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: manager-data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: omm-data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: goldendb-gtm-0 serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 annotations: "gdb.timezone": Asia/Shanghai env: - name: TZ value: Asia/Shanghai volumes: - name: localtime hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File replicas: 1 resources: limits: cpu: 1000m memory: 1Gi requests: cpu: 1000m memory: 1Gi volumeClaimTemplates: - name: data-agent spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-gtm spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi `kubectl apply -f test_create_goldendb-wcalnk.yaml` cluster.apps.kubeblocks.io/goldendb-wcalnk created apply test_create_goldendb-wcalnk.yaml Success `rm -rf test_create_goldendb-wcalnk.yaml` check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Creating Jun 19,2025 18:16 UTC+0800 clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1 / 1 1Gi / 1Gi data-agent:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:20 UTC+0800 data-gtm:10Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:10Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:10Gi omm-data:10Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1 / 1 1Gi / 1Gi insight-data:10Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb `kubectl get secrets -l app.kubernetes.io/instance=goldendb-wcalnk` set secret: goldendb-wcalnk-goldendb-dn-00-account-root `kubectl get secrets goldendb-wcalnk-goldendb-dn-00-account-root -o jsonpath="***.data.username***"` `kubectl get secrets goldendb-wcalnk-goldendb-dn-00-account-root -o jsonpath="***.data.password***"` `kubectl get secrets goldendb-wcalnk-goldendb-dn-00-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:db1x@NJ+1;DB_PORT:;DB_DATABASE: check pod goldendb-wcalnk-goldendb-dn-00-0 container_name dn exist password db1x@NJ+1 check pod goldendb-wcalnk-goldendb-dn-00-0 container_name lorry exist password db1x@NJ+1 No container logs contain secret password. describe cluster `kbcli cluster describe goldendb-wcalnk --namespace ns-gdyfe ` Name: goldendb-wcalnk Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-gdyfe goldendb Running WipeOut Endpoints: COMPONENT MODE INTERNAL EXTERNAL goldendb-cn-00 ReadWrite goldendb-wcalnk-goldendb-cn-00-proxy.ns-gdyfe.svc.cluster.local:8880 goldendb-zk ReadWrite goldendb-wcalnk-goldendb-zk.ns-gdyfe.svc.cluster.local:32181 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME goldendb-cn-00 goldendb-wcalnk-goldendb-cn-00-0 Running us-west-2a ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:20 UTC+0800 goldendb-cn-00 goldendb-wcalnk-goldendb-cn-00-1 Running us-west-2a ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:20 UTC+0800 goldendb-dn-00 goldendb-wcalnk-goldendb-dn-00-0 primary Running us-west-2a ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 goldendb-dn-00 goldendb-wcalnk-goldendb-dn-00-1 standby Running us-west-2a ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 goldendb-gtm-0 goldendb-wcalnk-goldendb-gtm-0-0 primary Running us-west-2a ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:20 UTC+0800 goldendb-manager goldendb-wcalnk-goldendb-manager-0 Running us-west-2a ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 goldendb-zk goldendb-wcalnk-goldendb-zk-0 Running us-west-2a ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS goldendb-cn-00 false 1 / 1 3Gi / 3Gi data-agent:10Gi kb-default-sc data-cn:10Gi kb-default-sc goldendb-dn-00 false 1 / 1 2Gi / 2Gi backup:10Gi kb-default-sc data-agent:10Gi kb-default-sc data-log-mysql-log:10Gi kb-default-sc data-mysql-data:10Gi kb-default-sc goldendb-gtm-0 false 1 / 1 1Gi / 1Gi data-agent:10Gi kb-default-sc data-gtm:10Gi kb-default-sc goldendb-manager false 3 / 3 8Gi / 8Gi insight-data:10Gi kb-default-sc manager-data:10Gi kb-default-sc omm-data:10Gi kb-default-sc goldendb-zk false 1 / 1 1Gi / 1Gi insight-data:10Gi kb-default-sc Images: COMPONENT TYPE IMAGE goldendb-cn-00 docker.io/apecloud/goldendb-cn:6.1.03.08 goldendb-dn-00 docker.io/apecloud/goldendb-dn:6.1.03.08 goldendb-gtm-0 docker.io/apecloud/goldendb-gtm:6.1.03.08 goldendb-manager docker.io/apecloud/goldendb-control:6.1.03.08 goldendb-zk docker.io/apecloud/goldendb-zk:6.1.03.08 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME backuprepo-kbcli-test Enabled 0 2 */1 * * full 7d Show cluster events: kbcli cluster list-events -n ns-gdyfe goldendb-wcalnk `kbcli cluster label goldendb-wcalnk app.kubernetes.io/instance- --namespace ns-gdyfe ` label "app.kubernetes.io/instance" not found. `kbcli cluster label goldendb-wcalnk app.kubernetes.io/instance=goldendb-wcalnk --namespace ns-gdyfe ` `kbcli cluster label goldendb-wcalnk --list --namespace ns-gdyfe ` NAME NAMESPACE LABELS goldendb-wcalnk ns-gdyfe app.kubernetes.io/instance=goldendb-wcalnk clusterdefinition.kubeblocks.io/name=goldendb clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=goldendb-wcalnk Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=goldendb-wcalnk --namespace ns-gdyfe ` `kbcli cluster label goldendb-wcalnk --list --namespace ns-gdyfe ` NAME NAMESPACE LABELS goldendb-wcalnk ns-gdyfe app.kubernetes.io/instance=goldendb-wcalnk case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=goldendb clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label goldendb-wcalnk case.name=kbcli.test2 --overwrite --namespace ns-gdyfe ` `kbcli cluster label goldendb-wcalnk --list --namespace ns-gdyfe ` NAME NAMESPACE LABELS goldendb-wcalnk ns-gdyfe app.kubernetes.io/instance=goldendb-wcalnk case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=goldendb clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label goldendb-wcalnk case.name- --namespace ns-gdyfe ` `kbcli cluster label goldendb-wcalnk --list --namespace ns-gdyfe ` NAME NAMESPACE LABELS goldendb-wcalnk ns-gdyfe app.kubernetes.io/instance=goldendb-wcalnk clusterdefinition.kubeblocks.io/name=goldendb clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect kbcli unsupported engine type: goldendb-dn-00 cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose goldendb-wcalnk --auto-approve --force=true --type vpc --enable true --components goldendb-dn-00 --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-expose-stjhc created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-expose-stjhc -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-expose-stjhc ns-gdyfe Expose goldendb-wcalnk goldendb-dn-00 Running 0/1 Jun 19,2025 18:27 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1 / 1 1Gi / 1Gi data-agent:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:20 UTC+0800 data-gtm:10Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:10Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:10Gi omm-data:10Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1 / 1 1Gi / 1Gi insight-data:10Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-expose-stjhc ns-gdyfe Expose goldendb-wcalnk goldendb-dn-00 Succeed 1/1 Jun 19,2025 18:27 UTC+0800 check ops status done ops_status:goldendb-wcalnk-expose-stjhc ns-gdyfe Expose goldendb-wcalnk goldendb-dn-00 Succeed 1/1 Jun 19,2025 18:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-expose-stjhc --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-expose-stjhc patched `kbcli cluster delete-ops --name goldendb-wcalnk-expose-stjhc --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-expose-stjhc deleted check component goldendb-zk exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-zk --namespace ns-gdyfe | (grep "goldendb-zk" || true )` check component goldendb-gtm-0 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-gtm-0 --namespace ns-gdyfe | (grep "goldendb-gtm-0" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale goldendb-wcalnk --auto-approve --force=true --components goldendb-zk,goldendb-gtm-0 --cpu 1100m --memory 1.1Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-wggqp created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-verticalscaling-wggqp -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-wggqp ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-zk,goldendb-gtm-0 Running 0/2 Jun 19,2025 18:28 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:10Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:10Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:10Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:10Gi omm-data:10Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-wggqp ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-zk,goldendb-gtm-0 Succeed 2/2 Jun 19,2025 18:28 UTC+0800 check ops status done ops_status:goldendb-wcalnk-verticalscaling-wggqp ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-zk,goldendb-gtm-0 Succeed 2/2 Jun 19,2025 18:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-verticalscaling-wggqp --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-verticalscaling-wggqp patched `kbcli cluster delete-ops --name goldendb-wcalnk-verticalscaling-wggqp --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-wggqp deleted test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-goldendb-wcalnk" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-goldendb-wcalnk.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-goldendb-wcalnk created apply test-chaos-mesh-podfailure-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-podfailure-goldendb-wcalnk.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:10Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:10Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:10Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:10Gi omm-data:10Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-goldendb-wcalnk" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover podfailure Success check component goldendb-manager exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-manager --namespace ns-gdyfe | (grep "goldendb-manager" || true )` `kubectl get pvc -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-manager,apps.kubeblocks.io/vct-name=insight-data,manager-data,omm-data --namespace ns-gdyfe ` No resources found in ns-gdyfe namespace. goldendb-wcalnk goldendb-manager insight-data,manager-data,omm-data pvc is empty cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in goldendb-wcalnk namespace. `kbcli cluster volume-expand goldendb-wcalnk --auto-approve --force=true --components goldendb-manager --volume-claim-templates insight-data,manager-data,omm-data --storage 15Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-gqbmf created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-volumeexpansion-gqbmf -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-gqbmf ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-manager Running 0/3 Jun 19,2025 18:33 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1 / 1 3Gi / 3Gi data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:20 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:10Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:10Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb No resources found in goldendb-wcalnk namespace. check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-gqbmf ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-manager Succeed 3/3 Jun 19,2025 18:33 UTC+0800 check ops status done ops_status:goldendb-wcalnk-volumeexpansion-gqbmf ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-manager Succeed 3/3 Jun 19,2025 18:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-volumeexpansion-gqbmf --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-volumeexpansion-gqbmf patched `kbcli cluster delete-ops --name goldendb-wcalnk-volumeexpansion-gqbmf --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-gqbmf deleted check component goldendb-cn-00 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-cn-00 --namespace ns-gdyfe | (grep "goldendb-cn-00" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale goldendb-wcalnk --auto-approve --force=true --components goldendb-cn-00 --cpu 1100m --memory 3.1Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-n6phb created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-verticalscaling-n6phb -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-n6phb ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-cn-00 Running 0/2 Jun 19,2025 18:34 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:10Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:10Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-n6phb ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-cn-00 Succeed 2/2 Jun 19,2025 18:34 UTC+0800 check ops status done ops_status:goldendb-wcalnk-verticalscaling-n6phb ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-cn-00 Succeed 2/2 Jun 19,2025 18:34 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-verticalscaling-n6phb --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-verticalscaling-n6phb patched `kbcli cluster delete-ops --name goldendb-wcalnk-verticalscaling-n6phb --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-n6phb deleted check component goldendb-gtm-0 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-gtm-0 --namespace ns-gdyfe | (grep "goldendb-gtm-0" || true )` `kubectl get pvc -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-gtm-0,apps.kubeblocks.io/vct-name=data-agent,data-gtm --namespace ns-gdyfe ` No resources found in ns-gdyfe namespace. goldendb-wcalnk goldendb-gtm-0 data-agent,data-gtm pvc is empty cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in goldendb-wcalnk namespace. `kbcli cluster volume-expand goldendb-wcalnk --auto-approve --force=true --components goldendb-gtm-0 --volume-claim-templates data-agent,data-gtm --storage 19Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-mgfcn created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-volumeexpansion-mgfcn -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-mgfcn ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-gtm-0 Running 0/2 Jun 19,2025 18:37 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb No resources found in goldendb-wcalnk namespace. check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-mgfcn ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-gtm-0 Succeed 2/2 Jun 19,2025 18:37 UTC+0800 check ops status done ops_status:goldendb-wcalnk-volumeexpansion-mgfcn ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-gtm-0 Succeed 2/2 Jun 19,2025 18:37 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-volumeexpansion-mgfcn --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-volumeexpansion-mgfcn patched `kbcli cluster delete-ops --name goldendb-wcalnk-volumeexpansion-mgfcn --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-mgfcn deleted test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-goldendb-wcalnk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-goldendb-wcalnk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-goldendb-wcalnk created apply test-chaos-mesh-dnsrandom-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-dnsrandom-goldendb-wcalnk.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-goldendb-wcalnk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover dnsrandom Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-goldendb-wcalnk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-goldendb-wcalnk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-goldendb-wcalnk created apply test-chaos-mesh-networkdelay-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-networkdelay-goldendb-wcalnk.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-goldendb-wcalnk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover networkdelay Success cluster update monitor KUBEBLOCKS NAMESPACE:kb-devqu get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster goldendb-wcalnk --namespace ns-gdyfe --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/goldendb-wcalnk patched check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb create curl-jq pod goldendb-wcalnk-kb-curl `kubectl run --namespace kb-devqu goldendb-wcalnk-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/goldendb-wcalnk-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE goldendb-wcalnk-kb-curl 0/1 ContainerCreating 0 1s check pod goldendb-wcalnk-kb-curl status done `kubectl wait --for=condition=Ready pod goldendb-wcalnk-kb-curl --namespace kb-devqu` pod/goldendb-wcalnk-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE goldendb-wcalnk-kb-curl 1/1 Running 0 6s kubectl exec -it goldendb-wcalnk-kb-curl --namespace kb-devqu -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it goldendb-wcalnk-kb-curl --namespace kb-devqu -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 34987 0 --:--:-- --:--:-- --:--:-- 40666 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750329868.522,"13.299999999999995"]***]*** k8s_pod_cpu_limit: 1750329868.522 monitor cluster Success `kubectl delete pod goldendb-wcalnk-kb-curl --namespace kb-devqu --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "goldendb-wcalnk-kb-curl" force deleted check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-goldendb-wcalnk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-1 action: partition mode: all target: mode: all selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-goldendb-wcalnk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-goldendb-wcalnk created apply test-chaos-mesh-networkpartition-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-networkpartition-goldendb-wcalnk.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-goldendb-wcalnk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover networkpartition Success cluster update enable-all-logs `kbcli cluster update goldendb-wcalnk --enable-all-logs=true --namespace ns-gdyfe ` error: failed to reconfigure log variables of target cluster: no config templates for component goldendb-cn-00 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb test failover delete pod:goldendb-wcalnk-goldendb-dn-00-1 `kubectl delete pod goldendb-wcalnk-goldendb-dn-00-1 --namespace ns-gdyfe ` pod "goldendb-wcalnk-goldendb-dn-00-1" deleted check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:47 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-0 failover Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-goldendb-wcalnk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-goldendb-wcalnk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-goldendb-wcalnk created apply test-chaos-mesh-networklossover-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-networklossover-goldendb-wcalnk.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:47 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-goldendb-wcalnk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover networklossover Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-goldendb-wcalnk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-goldendb-wcalnk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-goldendb-wcalnk created apply test-chaos-mesh-networkduplicate-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-networkduplicate-goldendb-wcalnk.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:22 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1 / 1 2Gi / 2Gi backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:47 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-goldendb-wcalnk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover networkduplicate Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale goldendb-wcalnk --auto-approve --force=true --components goldendb-dn-00 --cpu 1100m --memory 2.1Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-b2xwt created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-verticalscaling-b2xwt -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-b2xwt ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-dn-00 Running 0/2 Jun 19,2025 18:53 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:54 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:55 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-b2xwt ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-dn-00 Succeed 2/2 Jun 19,2025 18:53 UTC+0800 check ops status done ops_status:goldendb-wcalnk-verticalscaling-b2xwt ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-dn-00 Succeed 2/2 Jun 19,2025 18:53 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-verticalscaling-b2xwt --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-verticalscaling-b2xwt patched `kbcli cluster delete-ops --name goldendb-wcalnk-verticalscaling-b2xwt --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-b2xwt deleted test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-goldendb-wcalnk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-goldendb-wcalnk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-goldendb-wcalnk created apply test-chaos-mesh-fullcpu-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-fullcpu-goldendb-wcalnk.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:54 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 18:55 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-goldendb-wcalnk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover fullcpu Success test failover drainnode check node drain check node drain success kubectl get pod goldendb-wcalnk-goldendb-dn-00-1 --namespace ns-gdyfe -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-10-126.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod goldendb-wcalnk-goldendb-dn-00-0 --namespace ns-gdyfe -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-15-192.us-west-2.compute.internal success kubectl drain ip-172-31-10-126.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-10-126.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-tr4cj, kb-devqu/kb-addon-apecloud-otel-collector-v75cf, kube-system/aws-node-45rqf, kube-system/ebs-csi-node-4spbf, kube-system/kube-proxy-mnp8f evicting pod ns-gdyfe/goldendb-wcalnk-goldendb-dn-00-1 pod/goldendb-wcalnk-goldendb-dn-00-1 evicted node/ip-172-31-10-126.us-west-2.compute.internal drained kubectl uncordon ip-172-31-10-126.us-west-2.compute.internal node/ip-172-31-10-126.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:54 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:58 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-0 failover drainnode Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-goldendb-wcalnk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-goldendb-wcalnk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-goldendb-wcalnk created apply test-chaos-mesh-dnserror-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-dnserror-goldendb-wcalnk.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:54 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:58 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-goldendb-wcalnk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-0 failover dnserror Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-goldendb-wcalnk" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-goldendb-wcalnk.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-goldendb-wcalnk created apply test-chaos-mesh-timeoffset-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-timeoffset-goldendb-wcalnk.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 18:36 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 18:35 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 18:54 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-12-213.us-west-2.compute.internal/172.31.12.213 Jun 19,2025 18:58 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 18:28 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 18:17 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-goldendb-wcalnk" force deleted timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-goldendb-wcalnk patched check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-0 failover timeoffset Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart goldendb-wcalnk --auto-approve --force=true --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-restart-7s6j4 created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-restart-7s6j4 -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-restart-7s6j4 ns-gdyfe Restart goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Running 0/7 Jun 19,2025 19:04 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:10Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-restart-7s6j4 ns-gdyfe Restart goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Succeed 7/7 Jun 19,2025 19:04 UTC+0800 check ops status done ops_status:goldendb-wcalnk-restart-7s6j4 ns-gdyfe Restart goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Succeed 7/7 Jun 19,2025 19:04 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-restart-7s6j4 --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-restart-7s6j4 patched `kbcli cluster delete-ops --name goldendb-wcalnk-restart-7s6j4 --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-restart-7s6j4 deleted check component goldendb-zk exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-zk --namespace ns-gdyfe | (grep "goldendb-zk" || true )` `kubectl get pvc -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-zk,apps.kubeblocks.io/vct-name=insight-data --namespace ns-gdyfe ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in goldendb-wcalnk namespace. `kbcli cluster volume-expand goldendb-wcalnk --auto-approve --force=true --components goldendb-zk --volume-claim-templates insight-data --storage 13Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-sc4z4 created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-volumeexpansion-sc4z4 -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-sc4z4 ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-zk Running 0/1 Jun 19,2025 19:08 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb No resources found in goldendb-wcalnk namespace. check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-sc4z4 ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-zk Succeed 1/1 Jun 19,2025 19:08 UTC+0800 check ops status done ops_status:goldendb-wcalnk-volumeexpansion-sc4z4 ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-zk Succeed 1/1 Jun 19,2025 19:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-volumeexpansion-sc4z4 --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-volumeexpansion-sc4z4 patched `kbcli cluster delete-ops --name goldendb-wcalnk-volumeexpansion-sc4z4 --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-sc4z4 deleted test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote goldendb-wcalnk --auto-approve --force=true --component goldendb-dn-00 --instance goldendb-wcalnk-goldendb-dn-00-0 --namespace ns-gdyfe ` component:goldendb-dn-00 OpsRequest goldendb-wcalnk-switchover-t45xn created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-switchover-t45xn -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-switchover-t45xn ns-gdyfe Switchover goldendb-wcalnk Creating -/- Jun 19,2025 19:08 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-switchover-t45xn ns-gdyfe Switchover goldendb-wcalnk goldendb-dn-00 Succeed 1/1 Jun 19,2025 19:08 UTC+0800 check ops status done ops_status:goldendb-wcalnk-switchover-t45xn ns-gdyfe Switchover goldendb-wcalnk goldendb-dn-00 Succeed 1/1 Jun 19,2025 19:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-switchover-t45xn --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-switchover-t45xn patched `kbcli cluster delete-ops --name goldendb-wcalnk-switchover-t45xn --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-switchover-t45xn deleted switchover pod:goldendb-wcalnk-goldendb-dn-00-0 switchover success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-goldendb-wcalnk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-goldendb-wcalnk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-goldendb-wcalnk created apply test-chaos-mesh-networkbandwidthover-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-goldendb-wcalnk.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-goldendb-wcalnk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover networkbandwidthover Success check component goldendb-cn-00 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-cn-00 --namespace ns-gdyfe | (grep "goldendb-cn-00" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in goldendb-wcalnk namespace. `kbcli cluster hscale goldendb-wcalnk --auto-approve --force=true --components goldendb-cn-00 --replicas 3 --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-horizontalscaling-542b6 created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-horizontalscaling-542b6 -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-horizontalscaling-542b6 ns-gdyfe HorizontalScaling goldendb-wcalnk goldendb-cn-00 Running 0/1 Jun 19,2025 19:12 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster goldendb-wcalnk yaml-------------------------------------- `kubectl get cluster goldendb-wcalnk -o yaml --namespace ns-gdyfe ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T11:11:43.227051637Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"goldendb-wcalnk","namespace":"ns-gdyfe"***,"spec":***"clusterDefinitionRef":"goldendb","componentSpecs":[***"annotations":***"gdb.timezone":"Asia/Shanghai"***,"env":[***"name":"TZ","value":"Asia/Shanghai"***],"name":"goldendb-dn-00","replicas":2,"resources":***"limits":***"cpu":"1000m","memory":"2Gi"***,"requests":***"cpu":"1000m","memory":"2Gi"***,"serviceAccountName":"kb-goldendb-wcalnk","serviceVersion":"6.1.3","volumeClaimTemplates":[***"name":"backup","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"data-agent","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"data-log-mysql-log","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"data-mysql-data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***],"volumes":[***"hostPath":***"path":"/usr/share/zoneinfo/Asia/Shanghai","type":"File"***,"name":"localtime"***]***,***"annotations":***"gdb.timezone":"Asia/Shanghai"***,"env":[***"name":"TZ","value":"Asia/Shanghai"***],"name":"goldendb-cn-00","replicas":2,"resources":***"limits":***"cpu":"1000m","memory":"3Gi"***,"requests":***"cpu":"1000m","memory":"3Gi"***,"serviceAccountName":"kb-goldendb-wcalnk","serviceVersion":"6.1.3","volumeClaimTemplates":[***"name":"data-agent","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"data-cn","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***],"volumes":[***"hostPath":***"path":"/usr/share/zoneinfo/Asia/Shanghai","type":"File"***,"name":"localtime"***]***,***"annotations":***"gdb.timezone":"Asia/Shanghai"***,"env":[***"name":"TZ","value":"Asia/Shanghai"***],"name":"goldendb-zk","replicas":1,"resources":***"limits":***"cpu":"1000m","memory":"1Gi"***,"requests":***"cpu":"1000m","memory":"1Gi"***,"serviceAccountName":"kb-goldendb-wcalnk","serviceVersion":"6.1.3","volumeClaimTemplates":[***"name":"insight-data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***],"volumes":[***"hostPath":***"path":"/usr/share/zoneinfo/Asia/Shanghai","type":"File"***,"name":"localtime"***]***,***"annotations":***"gdb.timezone":"Asia/Shanghai"***,"env":[***"name":"TZ","value":"Asia/Shanghai"***],"name":"goldendb-manager","replicas":1,"resources":***"limits":***"cpu":"3000m","memory":"8Gi"***,"requests":***"cpu":"3000m","memory":"8Gi"***,"serviceAccountName":"kb-goldendb-wcalnk","serviceVersion":"6.1.3","volumeClaimTemplates":[***"name":"insight-data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"manager-data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"omm-data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***],"volumes":[***"hostPath":***"path":"/usr/share/zoneinfo/Asia/Shanghai","type":"File"***,"name":"localtime"***]***,***"annotations":***"gdb.timezone":"Asia/Shanghai"***,"env":[***"name":"TZ","value":"Asia/Shanghai"***],"name":"goldendb-gtm-0","replicas":1,"resources":***"limits":***"cpu":"1000m","memory":"1Gi"***,"requests":***"cpu":"1000m","memory":"1Gi"***,"serviceAccountName":"kb-goldendb-wcalnk","serviceVersion":"6.1.3","volumeClaimTemplates":[***"name":"data-agent","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***,***"name":"data-gtm","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"10Gi"***,"storageClassName":null***],"volumes":[***"hostPath":***"path":"/usr/share/zoneinfo/Asia/Shanghai","type":"File"***,"name":"localtime"***]***],"terminationPolicy":"WipeOut","topology":"goldendb-cluster"*** creationTimestamp: "2025-06-19T10:16:13Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 13 labels: app.kubernetes.io/instance: goldendb-wcalnk clusterdefinition.kubeblocks.io/name: goldendb clusterversion.kubeblocks.io/name: "" name: goldendb-wcalnk namespace: ns-gdyfe resourceVersion: "67737" uid: 7c2966f4-2ace-4ca9-9d27-c99c4e975b0b spec: clusterDefinitionRef: goldendb componentSpecs: - annotations: gdb.timezone: Asia/Shanghai componentDef: golden-cn-6.1 disableExporter: false env: - name: TZ value: Asia/Shanghai name: goldendb-cn-00 replicas: 3 resources: limits: cpu: 1100m memory: 3328599654400m requests: cpu: 1100m memory: 3328599654400m serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 volumeClaimTemplates: - name: data-agent spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-cn spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi volumes: - hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File name: localtime - annotations: gdb.timezone: Asia/Shanghai componentDef: golden-dn-6.1 env: - name: TZ value: Asia/Shanghai name: goldendb-dn-00 replicas: 2 resources: limits: cpu: 1100m memory: 2254857830400m requests: cpu: 1100m memory: 2254857830400m serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 volumeClaimTemplates: - name: backup spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-agent spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-log-mysql-log spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi - name: data-mysql-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi volumes: - hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File name: localtime - annotations: gdb.timezone: Asia/Shanghai componentDef: golden-gtm-6.1 env: - name: TZ value: Asia/Shanghai name: goldendb-gtm-0 replicas: 1 resources: limits: cpu: 1100m memory: 1181116006400m requests: cpu: 1100m memory: 1181116006400m serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 volumeClaimTemplates: - name: data-agent spec: accessModes: - ReadWriteOnce resources: requests: storage: 19Gi - name: data-gtm spec: accessModes: - ReadWriteOnce resources: requests: storage: 19Gi volumes: - hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File name: localtime - annotations: gdb.timezone: Asia/Shanghai componentDef: golden-manager-6.1 env: - name: TZ value: Asia/Shanghai name: goldendb-manager replicas: 1 resources: limits: cpu: "3" memory: 8Gi requests: cpu: "3" memory: 8Gi serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 volumeClaimTemplates: - name: insight-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 15Gi - name: manager-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 15Gi - name: omm-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 15Gi volumes: - hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File name: localtime - annotations: gdb.timezone: Asia/Shanghai componentDef: golden-zk-6.1 env: - name: TZ value: Asia/Shanghai name: goldendb-zk replicas: 1 resources: limits: cpu: 1100m memory: 1181116006400m requests: cpu: 1100m memory: 1181116006400m serviceAccountName: kb-goldendb-wcalnk serviceVersion: 6.1.3 volumeClaimTemplates: - name: insight-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 13Gi volumes: - hostPath: path: /usr/share/zoneinfo/Asia/Shanghai type: File name: localtime resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "true" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: goldendb-dn-00 name: goldendb-dn-00-vpc roleSelector: primary serviceName: goldendb-dn-00-vpc spec: ports: - name: dbagentport nodePort: 31617 port: 5018 protocol: TCP targetPort: dbagentport type: LoadBalancer storage: size: "0" terminationPolicy: WipeOut topology: goldendb-cluster status: clusterDefGeneration: 1 components: goldendb-cn-00: phase: Updating podsReady: false podsReadyTime: "2025-06-19T11:07:02Z" goldendb-dn-00: message: InstanceSet/goldendb-wcalnk-goldendb-dn-00: '["goldendb-wcalnk-goldendb-dn-00-0"]' phase: Running podsReady: true podsReadyTime: "2025-06-19T11:11:58Z" goldendb-gtm-0: phase: Running podsReady: true podsReadyTime: "2025-06-19T11:05:43Z" goldendb-manager: phase: Running podsReady: true podsReadyTime: "2025-06-19T11:06:25Z" goldendb-zk: phase: Running podsReady: true podsReadyTime: "2025-06-19T11:08:27Z" conditions: - lastTransitionTime: "2025-06-19T10:16:13Z" message: 'The operator has started the provisioning of Cluster: goldendb-wcalnk' observedGeneration: 13 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:16:13Z" message: Successfully applied for resources observedGeneration: 13 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:12:18Z" message: 'pods are not ready in Components: [goldendb-cn-00], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:12:18Z" message: 'pods are unavailable in Components: [goldendb-cn-00], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 13 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster goldendb-wcalnk-------------------------------------- `kubectl describe cluster goldendb-wcalnk --namespace ns-gdyfe ` Name: goldendb-wcalnk Namespace: ns-gdyfe Labels: app.kubernetes.io/instance=goldendb-wcalnk clusterdefinition.kubeblocks.io/name=goldendb clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-06-19T11:11:43.227051637Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:16:13Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 13 Resource Version: 67737 UID: 7c2966f4-2ace-4ca9-9d27-c99c4e975b0b Spec: Cluster Definition Ref: goldendb Component Specs: Annotations: gdb.timezone: Asia/Shanghai Component Def: golden-cn-6.1 Disable Exporter: false Env: Name: TZ Value: Asia/Shanghai Name: goldendb-cn-00 Replicas: 3 Resources: Limits: Cpu: 1100m Memory: 3328599654400m Requests: Cpu: 1100m Memory: 3328599654400m Service Account Name: kb-goldendb-wcalnk Service Version: 6.1.3 Volume Claim Templates: Name: data-agent Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 10Gi Name: data-cn Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 10Gi Volumes: Host Path: Path: /usr/share/zoneinfo/Asia/Shanghai Type: File Name: localtime Annotations: gdb.timezone: Asia/Shanghai Component Def: golden-dn-6.1 Env: Name: TZ Value: Asia/Shanghai Name: goldendb-dn-00 Replicas: 2 Resources: Limits: Cpu: 1100m Memory: 2254857830400m Requests: Cpu: 1100m Memory: 2254857830400m Service Account Name: kb-goldendb-wcalnk Service Version: 6.1.3 Volume Claim Templates: Name: backup Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 10Gi Name: data-agent Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 10Gi Name: data-log-mysql-log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 10Gi Name: data-mysql-data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 10Gi Volumes: Host Path: Path: /usr/share/zoneinfo/Asia/Shanghai Type: File Name: localtime Annotations: gdb.timezone: Asia/Shanghai Component Def: golden-gtm-6.1 Env: Name: TZ Value: Asia/Shanghai Name: goldendb-gtm-0 Replicas: 1 Resources: Limits: Cpu: 1100m Memory: 1181116006400m Requests: Cpu: 1100m Memory: 1181116006400m Service Account Name: kb-goldendb-wcalnk Service Version: 6.1.3 Volume Claim Templates: Name: data-agent Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 19Gi Name: data-gtm Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 19Gi Volumes: Host Path: Path: /usr/share/zoneinfo/Asia/Shanghai Type: File Name: localtime Annotations: gdb.timezone: Asia/Shanghai Component Def: golden-manager-6.1 Env: Name: TZ Value: Asia/Shanghai Name: goldendb-manager Replicas: 1 Resources: Limits: Cpu: 3 Memory: 8Gi Requests: Cpu: 3 Memory: 8Gi Service Account Name: kb-goldendb-wcalnk Service Version: 6.1.3 Volume Claim Templates: Name: insight-data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 15Gi Name: manager-data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 15Gi Name: omm-data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 15Gi Volumes: Host Path: Path: /usr/share/zoneinfo/Asia/Shanghai Type: File Name: localtime Annotations: gdb.timezone: Asia/Shanghai Component Def: golden-zk-6.1 Env: Name: TZ Value: Asia/Shanghai Name: goldendb-zk Replicas: 1 Resources: Limits: Cpu: 1100m Memory: 1181116006400m Requests: Cpu: 1100m Memory: 1181116006400m Service Account Name: kb-goldendb-wcalnk Service Version: 6.1.3 Volume Claim Templates: Name: insight-data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 13Gi Volumes: Host Path: Path: /usr/share/zoneinfo/Asia/Shanghai Type: File Name: localtime Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: true service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: goldendb-dn-00 Name: goldendb-dn-00-vpc Role Selector: primary Service Name: goldendb-dn-00-vpc Spec: Ports: Name: dbagentport Node Port: 31617 Port: 5018 Protocol: TCP Target Port: dbagentport Type: LoadBalancer Storage: Size: 0 Termination Policy: WipeOut Topology: goldendb-cluster Status: Cluster Def Generation: 1 Components: goldendb-cn-00: Phase: Updating Pods Ready: false Pods Ready Time: 2025-06-19T11:07:02Z goldendb-dn-00: Message: InstanceSet/goldendb-wcalnk-goldendb-dn-00: ["goldendb-wcalnk-goldendb-dn-00-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T11:11:58Z goldendb-gtm-0: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T11:05:43Z Goldendb - Manager: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T11:06:25Z Goldendb - Zk: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T11:08:27Z Conditions: Last Transition Time: 2025-06-19T10:16:13Z Message: The operator has started the provisioning of Cluster: goldendb-wcalnk Observed Generation: 13 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:16:13Z Message: Successfully applied for resources Observed Generation: 13 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:12:18Z Message: pods are not ready in Components: [goldendb-cn-00], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-06-19T11:12:18Z Message: pods are unavailable in Components: [goldendb-cn-00], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 13 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 60m (x5 over 60m) event-controller Pod goldendb-wcalnk-goldendb-manager-0: Readiness probe failed: dial tcp 172.31.8.49:8444: connect: connection refused Warning ComponentsNotReady 60m cluster-controller pods are unavailable in Components: [goldendb-cn-00 goldendb-gtm-0], refer to related component message in Cluster.status.components Warning ReplicasNotReady 60m cluster-controller pods are not ready in Components: [goldendb-cn-00 goldendb-gtm-0], refer to related component message in Cluster.status.components Warning ComponentsNotReady 58m cluster-controller pods are unavailable in Components: [goldendb-dn-00], refer to related component message in Cluster.status.components Warning ReplicasNotReady 58m cluster-controller pods are not ready in Components: [goldendb-dn-00], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 58m (x5 over 64m) cluster-controller component is Creating Warning Abnormal 55m cluster-controller Cluster: goldendb-wcalnk is Abnormal, check according to the components message Normal ClusterReady 54m cluster-controller Cluster: goldendb-wcalnk is ready, current phase is Running Normal Running 54m cluster-controller Cluster: goldendb-wcalnk is ready, current phase is Running Normal AllReplicasReady 54m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ApplyResourcesSucceed 53m (x3 over 64m) cluster-controller Successfully applied for resources Normal ComponentPhaseTransition 49m (x2 over 55m) cluster-controller component is Failed Warning Failed 49m event-controller Pod goldendb-wcalnk-goldendb-dn-00-0: Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown Warning BackOff 48m (x5 over 49m) event-controller Pod goldendb-wcalnk-goldendb-dn-00-0: Back-off restarting failed container dn in pod goldendb-wcalnk-goldendb-dn-00-0_ns-gdyfe(a7e34ba9-3bff-4b5b-b1de-20b5516d2b51) Warning Unhealthy 48m event-controller Pod goldendb-wcalnk-goldendb-dn-00-0: Readiness probe failed: Get "http://172.31.7.117:3501/v1.0/checkrole": dial tcp 172.31.7.117:3501: connect: connection refused Warning Unhealthy 39m (x7 over 40m) event-controller Pod goldendb-wcalnk-goldendb-dn-00-1: Readiness probe failed: Get "http://172.31.5.188:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal PreCheckSucceed 27m (x11 over 64m) cluster-controller The operator has started the provisioning of Cluster: goldendb-wcalnk Normal ComponentPhaseTransition 16m (x13 over 52m) cluster-controller component is Updating Warning Unhealthy 14m event-controller Pod goldendb-wcalnk-goldendb-manager-0: Readiness probe failed: dial tcp 172.31.8.63:8444: connect: connection refused Warning Unhealthy 9m26s (x12 over 30m) event-controller Pod goldendb-wcalnk-goldendb-dn-00-0: Readiness probe failed: Get "http://172.31.7.117:3501/v1.0/checkrole": dial tcp 172.31.7.117:3501: connect: no route to host Normal ComponentPhaseTransition 9m11s (x23 over 63m) cluster-controller component is Running Normal HorizontalScale 8m51s (x2 over 8m55s) component-controller start horizontal scale component goldendb-cn-00 of cluster goldendb-wcalnk from 2 to 3 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-2 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:12 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done No resources found in goldendb-wcalnk namespace. check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-horizontalscaling-542b6 ns-gdyfe HorizontalScaling goldendb-wcalnk goldendb-cn-00 Succeed 1/1 Jun 19,2025 19:12 UTC+0800 check ops status done ops_status:goldendb-wcalnk-horizontalscaling-542b6 ns-gdyfe HorizontalScaling goldendb-wcalnk goldendb-cn-00 Succeed 1/1 Jun 19,2025 19:12 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-horizontalscaling-542b6 --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-horizontalscaling-542b6 patched `kbcli cluster delete-ops --name goldendb-wcalnk-horizontalscaling-542b6 --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-horizontalscaling-542b6 deleted check component goldendb-cn-00 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-cn-00 --namespace ns-gdyfe | (grep "goldendb-cn-00" || true )` cluster hscale check cluster status before ops cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running No resources found in goldendb-wcalnk namespace. `kbcli cluster hscale goldendb-wcalnk --auto-approve --force=true --components goldendb-cn-00 --replicas 2 --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-horizontalscaling-kz6d4 created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-horizontalscaling-kz6d4 -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-horizontalscaling-kz6d4 ns-gdyfe HorizontalScaling goldendb-wcalnk goldendb-cn-00 Running 0/1 Jun 19,2025 19:23 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:10Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:10Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb No resources found in goldendb-wcalnk namespace. check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-horizontalscaling-kz6d4 ns-gdyfe HorizontalScaling goldendb-wcalnk goldendb-cn-00 Succeed 1/1 Jun 19,2025 19:23 UTC+0800 check ops status done ops_status:goldendb-wcalnk-horizontalscaling-kz6d4 ns-gdyfe HorizontalScaling goldendb-wcalnk goldendb-cn-00 Succeed 1/1 Jun 19,2025 19:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-horizontalscaling-kz6d4 --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-horizontalscaling-kz6d4 patched `kbcli cluster delete-ops --name goldendb-wcalnk-horizontalscaling-kz6d4 --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-horizontalscaling-kz6d4 deleted check component goldendb-cn-00 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-cn-00 --namespace ns-gdyfe | (grep "goldendb-cn-00" || true )` `kubectl get pvc -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-cn-00,apps.kubeblocks.io/vct-name=data-agent,data-cn --namespace ns-gdyfe ` No resources found in ns-gdyfe namespace. goldendb-wcalnk goldendb-cn-00 data-agent,data-cn pvc is empty cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in goldendb-wcalnk namespace. `kbcli cluster volume-expand goldendb-wcalnk --auto-approve --force=true --components goldendb-cn-00 --volume-claim-templates data-agent,data-cn --storage 16Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-htqct created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-volumeexpansion-htqct -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-htqct ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-cn-00 Running 0/4 Jun 19,2025 19:24 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3 / 3 8Gi / 8Gi insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:05 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb No resources found in goldendb-wcalnk namespace. check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-volumeexpansion-htqct ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-cn-00 Succeed 4/4 Jun 19,2025 19:24 UTC+0800 check ops status done ops_status:goldendb-wcalnk-volumeexpansion-htqct ns-gdyfe VolumeExpansion goldendb-wcalnk goldendb-cn-00 Succeed 4/4 Jun 19,2025 19:24 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-volumeexpansion-htqct --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-volumeexpansion-htqct patched `kbcli cluster delete-ops --name goldendb-wcalnk-volumeexpansion-htqct --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-volumeexpansion-htqct deleted check component goldendb-manager exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-manager --namespace ns-gdyfe | (grep "goldendb-manager" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale goldendb-wcalnk --auto-approve --force=true --components goldendb-manager --cpu 3100m --memory 8.1Gi --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-w2nvx created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-verticalscaling-w2nvx -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-w2nvx ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-manager Running 0/1 Jun 19,2025 19:25 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:06 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-2-115.us-west-2.compute.internal/172.31.2.115 Jun 19,2025 19:05 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:06 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:05 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:05 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:25 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:05 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-verticalscaling-w2nvx ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-manager Succeed 1/1 Jun 19,2025 19:25 UTC+0800 check ops status done ops_status:goldendb-wcalnk-verticalscaling-w2nvx ns-gdyfe VerticalScaling goldendb-wcalnk goldendb-manager Succeed 1/1 Jun 19,2025 19:25 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-verticalscaling-w2nvx --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-verticalscaling-w2nvx patched `kbcli cluster delete-ops --name goldendb-wcalnk-verticalscaling-w2nvx --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-verticalscaling-w2nvx deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop goldendb-wcalnk --auto-approve --force=true --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-stop-vrnxz created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-stop-vrnxz -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-stop-vrnxz ns-gdyfe Stop goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Running 0/7 Jun 19,2025 19:27 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Stopping Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-stop-vrnxz ns-gdyfe Stop goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Succeed 7/7 Jun 19,2025 19:27 UTC+0800 check ops status done ops_status:goldendb-wcalnk-stop-vrnxz ns-gdyfe Stop goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Succeed 7/7 Jun 19,2025 19:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-stop-vrnxz --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-stop-vrnxz patched `kbcli cluster delete-ops --name goldendb-wcalnk-stop-vrnxz --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-stop-vrnxz deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start goldendb-wcalnk --force=true --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-start-7zz9x created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-start-7zz9x -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-start-7zz9x ns-gdyfe Start goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Running 0/7 Jun 19,2025 19:29 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:29 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:29 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:29 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:29 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:29 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:29 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-start-7zz9x ns-gdyfe Start goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Succeed 7/7 Jun 19,2025 19:29 UTC+0800 check ops status done ops_status:goldendb-wcalnk-start-7zz9x ns-gdyfe Start goldendb-wcalnk goldendb-cn-00,goldendb-dn-00,goldendb-gtm-0,goldendb-manager,goldendb-zk Succeed 7/7 Jun 19,2025 19:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-start-7zz9x --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-start-7zz9x patched `kbcli cluster delete-ops --name goldendb-wcalnk-start-7zz9x --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-start-7zz9x deleted check component goldendb-dn-00 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-dn-00 --namespace ns-gdyfe | (grep "goldendb-dn-00" || true )` check component goldendb-cn-00 exists `kubectl get components -l app.kubernetes.io/instance=goldendb-wcalnk,apps.kubeblocks.io/component-name=goldendb-cn-00 --namespace ns-gdyfe | (grep "goldendb-cn-00" || true )` cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart goldendb-wcalnk --auto-approve --force=true --components goldendb-dn-00,goldendb-cn-00 --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-restart-wndtc created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-restart-wndtc -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-restart-wndtc ns-gdyfe Restart goldendb-wcalnk goldendb-dn-00,goldendb-cn-00 Running 0/4 Jun 19,2025 19:30 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:32 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:31 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:31 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:32 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:29 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:29 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-restart-wndtc ns-gdyfe Restart goldendb-wcalnk goldendb-dn-00,goldendb-cn-00 Succeed 4/4 Jun 19,2025 19:30 UTC+0800 check ops status done ops_status:goldendb-wcalnk-restart-wndtc ns-gdyfe Restart goldendb-wcalnk goldendb-dn-00,goldendb-cn-00 Succeed 4/4 Jun 19,2025 19:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-restart-wndtc --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-restart-wndtc patched `kbcli cluster delete-ops --name goldendb-wcalnk-restart-wndtc --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-restart-wndtc deleted test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-goldendb-wcalnk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-goldendb-wcalnk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-goldendb-wcalnk created apply test-chaos-mesh-networkcorruptover-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-goldendb-wcalnk.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:32 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:31 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:31 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:32 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:29 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:29 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-1 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-0 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-goldendb-wcalnk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-goldendb-wcalnk" not found check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-1 failover networkcorruptover Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-goldendb-wcalnk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-goldendb-wcalnk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-goldendb-wcalnk namespace: ns-gdyfe spec: selector: namespaces: - ns-gdyfe labelSelectors: apps.kubeblocks.io/pod-name: goldendb-wcalnk-goldendb-dn-00-1 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-goldendb-wcalnk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-goldendb-wcalnk created apply test-chaos-mesh-oom-goldendb-wcalnk.yaml Success `rm -rf test-chaos-mesh-oom-goldendb-wcalnk.yaml` check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:32 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:31 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:31 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:32 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:29 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:29 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-goldendb-wcalnk --namespace ns-gdyfe ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-goldendb-wcalnk" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-goldendb-wcalnk patched check failover pod name failover pod name:goldendb-wcalnk-goldendb-dn-00-0 failover oom Success cluster update terminationPolicy WipeOut `kbcli cluster update goldendb-wcalnk --termination-policy=WipeOut --namespace ns-gdyfe ` cluster.apps.kubeblocks.io/goldendb-wcalnk updated (no change) check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:32 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:31 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:31 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:32 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:29 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:29 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb cluster list-logs `kbcli cluster list-logs goldendb-wcalnk --namespace ns-gdyfe ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update goldendb-wcalnk --enable-all-logs=true --namespace ns-gdyfe Error from server (NotFound): pods "goldendb-wcalnk-goldendb-dn-00-0" not found cluster logs `kbcli cluster logs goldendb-wcalnk --tail 30 --namespace ns-gdyfe ` Defaulted container "gtm" out of: gtm, lorry, init-lorry (init) ++ readlink -f /goldendb_config/manager_container_metadata_folder/container_info.ini + linkf=/goldendb_config/manager_container_metadata_folder/..2025_06_19_11_29_12.2743982544/container_info.ini + cp -r /goldendb_config/manager_container_metadata_folder/..2025_06_19_11_29_12.2743982544/container_info.ini /goldendb_config/ + sed -i 's/^status.podip.*=.*/status.podip = goldendb-wcalnk-goldendb-gtm-0-0.ns-gdyfe/' /goldendb_config/container_info.ini + sed -i 's/^server_id.*=.*/server_id = 1/' /goldendb_config/container_info.ini + sed -i 's/^cluster_name.*=.*/cluster_name = goldendb-wcalnk-goldendb-gtm-0/' /goldendb_config/container_info.ini + chmod 755 /goldendb_config/container_info.ini ++ ls /opt/kb/scripts/dbagent_action.py /opt/kb/scripts/rdb_action.py /opt/kb/scripts/restore_dn.py + for file in $(ls /opt/kb/scripts/*.py) ++ readlink -f /opt/kb/scripts/dbagent_action.py + linkf=/opt/kb/scripts/..2025_06_19_11_29_12.919535452/dbagent_action.py ++ basename /opt/kb/scripts/dbagent_action.py + baseName=dbagent_action.py + [[ dbagent_action.py == \r\e\s\t\o\r\e\_\d\n\.\p\y ]] + cp -f /opt/kb/scripts/..2025_06_19_11_29_12.919535452/dbagent_action.py /goldendb_config/containerStartup/ + for file in $(ls /opt/kb/scripts/*.py) ++ readlink -f /opt/kb/scripts/rdb_action.py + linkf=/opt/kb/scripts/..2025_06_19_11_29_12.919535452/rdb_action.py ++ basename /opt/kb/scripts/rdb_action.py + baseName=rdb_action.py + [[ rdb_action.py == \r\e\s\t\o\r\e\_\d\n\.\p\y ]] + cp -f /opt/kb/scripts/..2025_06_19_11_29_12.919535452/rdb_action.py /goldendb_config/containerStartup/ + for file in $(ls /opt/kb/scripts/*.py) ++ readlink -f /opt/kb/scripts/restore_dn.py + linkf=/opt/kb/scripts/..2025_06_19_11_29_12.919535452/restore_dn.py ++ basename /opt/kb/scripts/restore_dn.py + baseName=restore_dn.py + [[ restore_dn.py == \r\e\s\t\o\r\e\_\d\n\.\p\y ]] + [[ -d /home/zxdb/bin/restore_dn ]] + continue cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose goldendb-wcalnk --auto-approve --force=true --type vpc --enable false --components goldendb-dn-00 --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-expose-bgbkz created successfully, you can view the progress: kbcli cluster describe-ops goldendb-wcalnk-expose-bgbkz -n ns-gdyfe check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-expose-bgbkz ns-gdyfe Expose goldendb-wcalnk goldendb-dn-00 Running 0/1 Jun 19,2025 19:37 UTC+0800 check cluster status `kbcli cluster list goldendb-wcalnk --show-labels --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS goldendb-wcalnk ns-gdyfe goldendb WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=goldendb-wcalnk,clusterdefinition.kubeblocks.io/name=goldendb,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances goldendb-wcalnk --namespace ns-gdyfe ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME goldendb-wcalnk-goldendb-cn-00-0 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-0-176.us-west-2.compute.internal/172.31.0.176 Jun 19,2025 19:32 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-cn-00-1 ns-gdyfe goldendb-wcalnk goldendb-cn-00 Running us-west-2a 1100m / 1100m 3328599654400m / 3328599654400m data-agent:16Gi ip-172-31-13-158.us-west-2.compute.internal/172.31.13.158 Jun 19,2025 19:31 UTC+0800 data-cn:16Gi goldendb-wcalnk-goldendb-dn-00-0 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running primary us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-15-192.us-west-2.compute.internal/172.31.15.192 Jun 19,2025 19:31 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-dn-00-1 ns-gdyfe goldendb-wcalnk goldendb-dn-00 Running standby us-west-2a 1100m / 1100m 2254857830400m / 2254857830400m backup:10Gi ip-172-31-10-126.us-west-2.compute.internal/172.31.10.126 Jun 19,2025 19:32 UTC+0800 data-agent:10Gi data-log-mysql-log:10Gi data-mysql-data:10Gi goldendb-wcalnk-goldendb-gtm-0-0 ns-gdyfe goldendb-wcalnk goldendb-gtm-0 Running primary us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m data-agent:19Gi ip-172-31-6-226.us-west-2.compute.internal/172.31.6.226 Jun 19,2025 19:29 UTC+0800 data-gtm:19Gi goldendb-wcalnk-goldendb-manager-0 ns-gdyfe goldendb-wcalnk goldendb-manager Running us-west-2a 3100m / 3100m 8697308774400m / 8697308774400m insight-data:15Gi ip-172-31-2-199.us-west-2.compute.internal/172.31.2.199 Jun 19,2025 19:29 UTC+0800 manager-data:15Gi omm-data:15Gi goldendb-wcalnk-goldendb-zk-0 ns-gdyfe goldendb-wcalnk goldendb-zk Running us-west-2a 1100m / 1100m 1181116006400m / 1181116006400m insight-data:13Gi ip-172-31-5-67.us-west-2.compute.internal/172.31.5.67 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done primary: goldendb-wcalnk-goldendb-dn-00-0 goldendb-wcalnk-goldendb-gtm-0-0;standby: goldendb-wcalnk-goldendb-dn-00-1 connect unsupported engine type: goldendb check ops status `kbcli cluster list-ops goldendb-wcalnk --status all --namespace ns-gdyfe ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME goldendb-wcalnk-expose-bgbkz ns-gdyfe Expose goldendb-wcalnk goldendb-dn-00 Succeed 1/1 Jun 19,2025 19:37 UTC+0800 check ops status done ops_status:goldendb-wcalnk-expose-bgbkz ns-gdyfe Expose goldendb-wcalnk goldendb-dn-00 Succeed 1/1 Jun 19,2025 19:37 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests goldendb-wcalnk-expose-bgbkz --namespace ns-gdyfe ` opsrequest.apps.kubeblocks.io/goldendb-wcalnk-expose-bgbkz patched `kbcli cluster delete-ops --name goldendb-wcalnk-expose-bgbkz --force --auto-approve --namespace ns-gdyfe ` OpsRequest goldendb-wcalnk-expose-bgbkz deleted delete cluster goldendb-wcalnk `kbcli cluster delete goldendb-wcalnk --auto-approve --namespace ns-gdyfe ` Cluster goldendb-wcalnk deleted pod_info:goldendb-wcalnk-goldendb-cn-00-0 2/2 Running 0 5m21s goldendb-wcalnk-goldendb-cn-00-1 2/2 Running 0 6m32s goldendb-wcalnk-goldendb-dn-00-0 2/2 Running 0 6m32s goldendb-wcalnk-goldendb-dn-00-1 2/2 Running 1 (101s ago) 5m24s goldendb-wcalnk-goldendb-gtm-0-0 2/2 Running 0 8m51s goldendb-wcalnk-goldendb-manager-0 2/2 Running 0 8m51s goldendb-wcalnk-goldendb-zk-0 1/1 Running 0 8m53s pod_info:goldendb-wcalnk-goldendb-cn-00-0 2/2 Terminating 0 5m42s goldendb-wcalnk-goldendb-cn-00-1 2/2 Terminating 0 6m53s goldendb-wcalnk-goldendb-dn-00-0 1/2 Terminating 0 6m53s goldendb-wcalnk-goldendb-dn-00-1 1/2 Terminating 1 (2m2s ago) 5m45s goldendb-wcalnk-goldendb-gtm-0-0 1/2 Terminating 0 9m12s goldendb-wcalnk-goldendb-manager-0 2/2 Terminating 0 9m12s goldendb-wcalnk-goldendb-zk-0 1/1 Terminating 0 9m14s No resources found in ns-gdyfe namespace. delete cluster pod done No resources found in ns-gdyfe namespace. check cluster resource non-exist OK: pvc No resources found in ns-gdyfe namespace. delete cluster done No resources found in ns-gdyfe namespace. No resources found in ns-gdyfe namespace. No resources found in ns-gdyfe namespace. GoldenDB Test Suite All Done! --------------------------------------GoldenDB (Topology = goldendb-cluster Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=goldendb;]|[Description=Create a cluster with the specified cluster definition goldendb] [PASSED]|[Expose]|[Enable=true;TYPE=vpc;ComponentName=goldendb-dn-00]|[Description=Expose Enable the vpc service with goldendb-dn-00 component] [PASSED]|[VerticalScaling]|[ComponentName=goldendb-zk,goldendb-gtm-0]|[Description=VerticalScaling the cluster specify component goldendb-zk,goldendb-gtm-0] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[VolumeExpansion]|[ComponentName=goldendb-manager]|[Description=VolumeExpansion the cluster specify component goldendb-manager] [PASSED]|[VerticalScaling]|[ComponentName=goldendb-cn-00]|[Description=VerticalScaling the cluster specify component goldendb-cn-00] [PASSED]|[VolumeExpansion]|[ComponentName=goldendb-gtm-0]|[Description=VolumeExpansion the cluster specify component goldendb-gtm-0] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Network Delay Failover;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[VerticalScaling]|[ComponentName=goldendb-dn-00]|[Description=VerticalScaling the cluster specify component goldendb-dn-00] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=goldendb-zk]|[Description=VolumeExpansion the cluster specify component goldendb-zk] [PASSED]|[SwitchOver]|[ComponentName=goldendb-dn-00]|[Description=SwitchOver the cluster specify component goldendb-dn-00] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[HorizontalScaling Out]|[ComponentName=goldendb-cn-00]|[Description=HorizontalScaling Out the cluster specify component goldendb-cn-00] [PASSED]|[HorizontalScaling In]|[ComponentName=goldendb-cn-00]|[Description=HorizontalScaling In the cluster specify component goldendb-cn-00] [PASSED]|[VolumeExpansion]|[ComponentName=goldendb-cn-00]|[Description=VolumeExpansion the cluster specify component goldendb-cn-00] [PASSED]|[VerticalScaling]|[ComponentName=goldendb-manager]|[Description=VerticalScaling the cluster specify component goldendb-manager] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Restart]|[ComponentName=goldendb-dn-00,goldendb-cn-00]|[Description=Restart the cluster specify component goldendb-dn-00,goldendb-cn-00] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=goldendb-dn-00]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Expose]|[Disable=true;TYPE=vpc;ComponentName=goldendb-dn-00]|[Description=Expose Disable the vpc service with goldendb-dn-00 component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]