source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-smhfl ` `kubectl create namespace ns-smhfl` namespace/ns-smhfl created create namespace ns-smhfl done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 119M 0 --:--:-- --:--:-- --:--:-- 119M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:gms set component version No resources found no component version found unsupported component definition not found component version set replicas first:3 set replicas third:3 set minimum cmpv service version set minimum cmpv service version replicas:3 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Not found topology in cluster definition polardbx LIMIT_CPU:0.5 LIMIT_MEMORY:1 storage size: 5 No resources found in ns-smhfl namespace. termination_policy:WipeOut create 3 replica WipeOut polardbx cluster check cluster version check cluster definition apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: px-qji namespace: ns-smhfl spec: clusterDefinitionRef: polardbx clusterVersionRef: polardbx-v2.3 terminationPolicy: WipeOut componentSpecs: - name: gms componentDefRef: gms replicas: 3 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 500m memory: 1Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi - name: dn componentDefRef: dn replicas: 3 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 500m memory: 1Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi - name: cn componentDefRef: cn replicas: 1 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 500m memory: 1Gi - name: cdc componentDefRef: cdc replicas: 1 resources: requests: cpu: 500m memory: 1Gi limits: cpu: 500m memory: 1Gi `kubectl apply -f test_create_px-qji.yaml` cluster.apps.kubeblocks.io/px-qji created apply test_create_px-qji.yaml Success `rm -rf test_create_px-qji.yaml` check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Jun 19,2025 18:15 UTC+0800 clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 500m / 500m 1Gi / 1Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:15 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 500m / 500m 1Gi / 1Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:15 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:15 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:17 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:18 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:15 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-9-241.us-west-2.compute.internal/172.31.9.241 Jun 19,2025 18:17 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-6-202.us-west-2.compute.internal/172.31.6.202 Jun 19,2025 18:18 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check pod px-qji-gms-0 container_name engine exist password gkjs9gfk check pod px-qji-gms-0 container_name exporter exist password gkjs9gfk check pod px-qji-gms-0 container_name lorry exist password gkjs9gfk No container logs contain secret password. describe cluster `kbcli cluster describe px-qji --namespace ns-smhfl ` Name: px-qji Created Time: Jun 19,2025 18:15 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-smhfl polardbx polardbx-v2.3 Running WipeOut Endpoints: COMPONENT MODE INTERNAL EXTERNAL gms ReadWrite px-qji-gms.ns-smhfl.svc.cluster.local:3306 px-qji-gms.ns-smhfl.svc.cluster.local:9104 dn ReadWrite px-qji-dn.ns-smhfl.svc.cluster.local:3306 cn ReadWrite px-qji-cn.ns-smhfl.svc.cluster.local:3306 px-qji-cn.ns-smhfl.svc.cluster.local:9104 cdc ReadWrite px-qji-cdc.ns-smhfl.svc.cluster.local:3306 px-qji-cdc.ns-smhfl.svc.cluster.local:9104 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME cdc px-qji-cdc-0 Running us-west-2a ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:15 UTC+0800 cn px-qji-cn-0 Running us-west-2a ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:15 UTC+0800 dn px-qji-dn-0 leader Running us-west-2a ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:15 UTC+0800 dn px-qji-dn-1 follower Running us-west-2a ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:17 UTC+0800 dn px-qji-dn-2 follower Running us-west-2a ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:18 UTC+0800 gms px-qji-gms-0 follower Running us-west-2a ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:15 UTC+0800 gms px-qji-gms-1 leader Running us-west-2a ip-172-31-9-241.us-west-2.compute.internal/172.31.9.241 Jun 19,2025 18:17 UTC+0800 gms px-qji-gms-2 follower Running us-west-2a ip-172-31-6-202.us-west-2.compute.internal/172.31.6.202 Jun 19,2025 18:18 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS gms false 500m / 500m 1Gi / 1Gi data:5Gi kb-default-sc dn false 500m / 500m 1Gi / 1Gi data:5Gi kb-default-sc cn false 500m / 500m 1Gi / 1Gi cdc false 500m / 500m 1Gi / 1Gi Images: COMPONENT TYPE IMAGE gms gms docker.io/apecloud/polardbx-engine-2.0:80-8.0.18-20231101115000 dn dn docker.io/apecloud/polardbx-engine-2.0:80-8.0.18-20231101115000 cn cn docker.io/apecloud/polardbx-sql:5.4.18-20231101115000 cdc cdc docker.io/apecloud/polardbx-cdc:5.4.18-20231101115000 Show cluster events: kbcli cluster list-events -n ns-smhfl px-qji `kbcli cluster label px-qji app.kubernetes.io/instance- --namespace ns-smhfl ` label "app.kubernetes.io/instance" not found. `kbcli cluster label px-qji app.kubernetes.io/instance=px-qji --namespace ns-smhfl ` `kbcli cluster label px-qji --list --namespace ns-smhfl ` NAME NAMESPACE LABELS px-qji ns-smhfl app.kubernetes.io/instance=px-qji clusterdefinition.kubeblocks.io/name=polardbx clusterversion.kubeblocks.io/name=polardbx-v2.3 label cluster app.kubernetes.io/instance=px-qji Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=px-qji --namespace ns-smhfl ` `kbcli cluster label px-qji --list --namespace ns-smhfl ` NAME NAMESPACE LABELS px-qji ns-smhfl app.kubernetes.io/instance=px-qji case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=polardbx clusterversion.kubeblocks.io/name=polardbx-v2.3 label cluster case.name=kbcli.test1 Success `kbcli cluster label px-qji case.name=kbcli.test2 --overwrite --namespace ns-smhfl ` `kbcli cluster label px-qji --list --namespace ns-smhfl ` NAME NAMESPACE LABELS px-qji ns-smhfl app.kubernetes.io/instance=px-qji case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=polardbx clusterversion.kubeblocks.io/name=polardbx-v2.3 label cluster case.name=kbcli.test2 Success `kbcli cluster label px-qji case.name- --namespace ns-smhfl ` `kbcli cluster label px-qji --list --namespace ns-smhfl ` NAME NAMESPACE LABELS px-qji ns-smhfl app.kubernetes.io/instance=px-qji clusterdefinition.kubeblocks.io/name=polardbx clusterversion.kubeblocks.io/name=polardbx-v2.3 delete cluster label case.name Success 7 check component cn exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cn --namespace ns-smhfl | (grep "cn" || true )` check component cdc exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cdc --namespace ns-smhfl | (grep "cdc" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale px-qji --auto-approve --force=true --components cn,cdc --cpu 600m --memory 1.1Gi --namespace ns-smhfl ` OpsRequest px-qji-verticalscaling-phstd created successfully, you can view the progress: kbcli cluster describe-ops px-qji-verticalscaling-phstd -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-verticalscaling-phstd ns-smhfl VerticalScaling px-qji cn,cdc Running 0/2 Jun 19,2025 18:23 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:24 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:23 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:15 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:17 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:18 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:15 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-9-241.us-west-2.compute.internal/172.31.9.241 Jun 19,2025 18:17 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-6-202.us-west-2.compute.internal/172.31.6.202 Jun 19,2025 18:18 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-verticalscaling-phstd ns-smhfl VerticalScaling px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:23 UTC+0800 check ops status done ops_status:px-qji-verticalscaling-phstd ns-smhfl VerticalScaling px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-verticalscaling-phstd --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-verticalscaling-phstd patched `kbcli cluster delete-ops --name px-qji-verticalscaling-phstd --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-verticalscaling-phstd deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop px-qji --auto-approve --force=true --namespace ns-smhfl ` OpsRequest px-qji-stop-w7f8j created successfully, you can view the progress: kbcli cluster describe-ops px-qji-stop-w7f8j -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-stop-w7f8j ns-smhfl Stop px-qji cdc,cn,dn,gms Running 0/8 Jun 19,2025 18:25 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Stopping Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-stop-w7f8j ns-smhfl Stop px-qji cdc,cn,dn,gms Succeed 8/8 Jun 19,2025 18:25 UTC+0800 check ops status done ops_status:px-qji-stop-w7f8j ns-smhfl Stop px-qji cdc,cn,dn,gms Succeed 8/8 Jun 19,2025 18:25 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-stop-w7f8j --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-stop-w7f8j patched `kbcli cluster delete-ops --name px-qji-stop-w7f8j --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-stop-w7f8j deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start px-qji --force=true --namespace ns-smhfl ` OpsRequest px-qji-start-k2wkb created successfully, you can view the progress: kbcli cluster describe-ops px-qji-start-k2wkb -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-start-k2wkb ns-smhfl Start px-qji Running -/- Jun 19,2025 18:27 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:27 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:27 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:5Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-start-k2wkb ns-smhfl Start px-qji cdc,cn,dn,gms Succeed 8/8 Jun 19,2025 18:27 UTC+0800 check ops status done ops_status:px-qji-start-k2wkb ns-smhfl Start px-qji cdc,cn,dn,gms Succeed 8/8 Jun 19,2025 18:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-start-k2wkb --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-start-k2wkb patched `kbcli cluster delete-ops --name px-qji-start-k2wkb --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-start-k2wkb deleted check component gms exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=gms --namespace ns-smhfl | (grep "gms" || true )` check component dn exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=dn --namespace ns-smhfl | (grep "dn" || true )` `kubectl get pvc -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=gms,dn,apps.kubeblocks.io/vct-name=data --namespace ns-smhfl ` No resources found in ns-smhfl namespace. px-qji gms,dn data pvc is empty cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in px-qji namespace. `kbcli cluster volume-expand px-qji --auto-approve --force=true --components gms,dn --volume-claim-templates data --storage 9Gi --namespace ns-smhfl ` OpsRequest px-qji-volumeexpansion-7ss4s created successfully, you can view the progress: kbcli cluster describe-ops px-qji-volumeexpansion-7ss4s -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-volumeexpansion-7ss4s ns-smhfl VolumeExpansion px-qji gms,dn Running 0/6 Jun 19,2025 18:32 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:27 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:27 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done No resources found in px-qji namespace. check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-volumeexpansion-7ss4s ns-smhfl VolumeExpansion px-qji gms,dn Succeed 6/6 Jun 19,2025 18:32 UTC+0800 check ops status done ops_status:px-qji-volumeexpansion-7ss4s ns-smhfl VolumeExpansion px-qji gms,dn Succeed 6/6 Jun 19,2025 18:32 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-volumeexpansion-7ss4s --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-volumeexpansion-7ss4s patched `kbcli cluster delete-ops --name px-qji-volumeexpansion-7ss4s --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-volumeexpansion-7ss4s deleted cluster update monitor KUBEBLOCKS NAMESPACE:kb-devqu get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster px-qji --namespace ns-smhfl --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/px-qji patched check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:27 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:27 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done create curl-jq pod px-qji-kb-curl `kubectl run --namespace kb-devqu px-qji-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/px-qji-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE px-qji-kb-curl 0/1 ContainerCreating 0 1s check pod px-qji-kb-curl status done `kubectl wait --for=condition=Ready pod px-qji-kb-curl --namespace kb-devqu` pod/px-qji-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE px-qji-kb-curl 1/1 Running 0 6s kubectl exec -it px-qji-kb-curl --namespace kb-devqu -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it px-qji-kb-curl --namespace kb-devqu -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 31241 0 --:--:-- --:--:-- --:--:-- 40666 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750329280.234,"19.300000000000004"]***]*** k8s_pod_cpu_limit: 1750329280.234 monitor cluster Success `kubectl delete pod px-qji-kb-curl --namespace kb-devqu --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "px-qji-kb-curl" force deleted check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:27 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:27 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done check component cn exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cn --namespace ns-smhfl | (grep "cn" || true )` check component cdc exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cdc --namespace ns-smhfl | (grep "cdc" || true )` cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart px-qji --auto-approve --force=true --components cn,cdc --namespace ns-smhfl ` OpsRequest px-qji-restart-ngf5t created successfully, you can view the progress: kbcli cluster describe-ops px-qji-restart-ngf5t -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-restart-ngf5t ns-smhfl Restart px-qji cn,cdc Running 0/2 Jun 19,2025 18:35 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:35 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:35 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-restart-ngf5t ns-smhfl Restart px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:35 UTC+0800 check ops status done ops_status:px-qji-restart-ngf5t ns-smhfl Restart px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-restart-ngf5t --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-restart-ngf5t patched `kbcli cluster delete-ops --name px-qji-restart-ngf5t --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-restart-ngf5t deleted check component cn exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cn --namespace ns-smhfl | (grep "cn" || true )` check component cdc exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cdc --namespace ns-smhfl | (grep "cdc" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in px-qji namespace. `kbcli cluster hscale px-qji --auto-approve --force=true --components cn,cdc --replicas 2 --namespace ns-smhfl ` OpsRequest px-qji-horizontalscaling-58wxf created successfully, you can view the progress: kbcli cluster describe-ops px-qji-horizontalscaling-58wxf -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-horizontalscaling-58wxf ns-smhfl HorizontalScaling px-qji cn,cdc Running 0/2 Jun 19,2025 18:36 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Updating Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:35 UTC+0800 px-qji-cdc-1 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-1-21.us-west-2.compute.internal/172.31.1.21 Jun 19,2025 18:36 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:35 UTC+0800 px-qji-cn-1 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:36 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done No resources found in px-qji namespace. check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-horizontalscaling-58wxf ns-smhfl HorizontalScaling px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:36 UTC+0800 check ops status done ops_status:px-qji-horizontalscaling-58wxf ns-smhfl HorizontalScaling px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:36 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-horizontalscaling-58wxf --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-horizontalscaling-58wxf patched `kbcli cluster delete-ops --name px-qji-horizontalscaling-58wxf --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-horizontalscaling-58wxf deleted check component cn exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cn --namespace ns-smhfl | (grep "cn" || true )` check component cdc exists `kubectl get components -l app.kubernetes.io/instance=px-qji,apps.kubeblocks.io/component-name=cdc --namespace ns-smhfl | (grep "cdc" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in px-qji namespace. `kbcli cluster hscale px-qji --auto-approve --force=true --components cn,cdc --replicas 1 --namespace ns-smhfl ` OpsRequest px-qji-horizontalscaling-xt8m8 created successfully, you can view the progress: kbcli cluster describe-ops px-qji-horizontalscaling-xt8m8 -n ns-smhfl check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-horizontalscaling-xt8m8 ns-smhfl HorizontalScaling px-qji cn,cdc Running 0/2 Jun 19,2025 18:38 UTC+0800 check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:35 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:35 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done No resources found in px-qji namespace. check ops status `kbcli cluster list-ops px-qji --status all --namespace ns-smhfl ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME px-qji-horizontalscaling-xt8m8 ns-smhfl HorizontalScaling px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:38 UTC+0800 check ops status done ops_status:px-qji-horizontalscaling-xt8m8 ns-smhfl HorizontalScaling px-qji cn,cdc Succeed 2/2 Jun 19,2025 18:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests px-qji-horizontalscaling-xt8m8 --namespace ns-smhfl ` opsrequest.apps.kubeblocks.io/px-qji-horizontalscaling-xt8m8 patched `kbcli cluster delete-ops --name px-qji-horizontalscaling-xt8m8 --force --auto-approve --namespace ns-smhfl ` OpsRequest px-qji-horizontalscaling-xt8m8 deleted cluster update terminationPolicy WipeOut `kbcli cluster update px-qji --termination-policy=WipeOut --namespace ns-smhfl ` cluster.apps.kubeblocks.io/px-qji updated (no change) check cluster status `kbcli cluster list px-qji --show-labels --namespace ns-smhfl ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS px-qji ns-smhfl polardbx polardbx-v2.3 WipeOut Running Jun 19,2025 18:15 UTC+0800 app.kubernetes.io/instance=px-qji,clusterdefinition.kubeblocks.io/name=polardbx,clusterversion.kubeblocks.io/name=polardbx-v2.3 check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances px-qji --namespace ns-smhfl ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME px-qji-cdc-0 ns-smhfl px-qji cdc Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-10-241.us-west-2.compute.internal/172.31.10.241 Jun 19,2025 18:35 UTC+0800 px-qji-cn-0 ns-smhfl px-qji cn Running us-west-2a 600m / 600m 1181116006400m / 1181116006400m ip-172-31-7-169.us-west-2.compute.internal/172.31.7.169 Jun 19,2025 18:35 UTC+0800 px-qji-dn-0 ns-smhfl px-qji dn Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-9-102.us-west-2.compute.internal/172.31.9.102 Jun 19,2025 18:27 UTC+0800 px-qji-dn-1 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-21.us-west-2.compute.internal/172.31.12.21 Jun 19,2025 18:28 UTC+0800 px-qji-dn-2 ns-smhfl px-qji dn Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:30 UTC+0800 px-qji-gms-0 ns-smhfl px-qji gms Running leader us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-5-157.us-west-2.compute.internal/172.31.5.157 Jun 19,2025 18:27 UTC+0800 px-qji-gms-1 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-15-205.us-west-2.compute.internal/172.31.15.205 Jun 19,2025 18:28 UTC+0800 px-qji-gms-2 ns-smhfl px-qji gms Running follower us-west-2a 500m / 500m 1Gi / 1Gi data:9Gi ip-172-31-12-148.us-west-2.compute.internal/172.31.12.148 Jun 19,2025 18:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=px-qji` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets px-qji-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:polardbx_root;DB_PASSWORD:gkjs9gfk;DB_PORT:3306;DB_DATABASE: check cluster connect ` echo ' echo "select 1;" | mysql -hpx-qji-dn.ns-smhfl.svc.cluster.local -P3306 -upolardbx_root -p"gkjs9gfk"' | kubectl exec -it px-qji-dn-0 --namespace ns-smhfl -- sh ` check cluster connect done cluster list-logs `kbcli cluster list-logs px-qji --namespace ns-smhfl ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update px-qji --enable-all-logs=true --namespace ns-smhfl Error from server (NotFound): pods "px-qji-gms-0" not found cluster logs `kbcli cluster logs px-qji --tail 30 --namespace ns-smhfl ` Defaulted container "engine" out of: engine, exporter, lorry, tools-updater (init), init-lorry (init) /tools/xstore/current/venv/lib64/python3.6/site-packages/pymysql/_auth.py:8: CryptographyDeprecationWarning: Python 3.6 is no longer supported by the Python core team. Therefore, support for it is deprecated in cryptography. The next release of cryptography will remove support for Python 3.6. from cryptography.hazmat.backends import default_backend 2025-06-19 10:28:09,359 - AuditController - INFO - Creating mysql connection pool... 2025-06-19 10:28:09,373 - AuditController - ERROR - Creating mysql connection pool failed: 2003: Can't connect to MySQL server on 'localhost:3306' (111 Connection refused) wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) 2025-06-19 10:28:14,378 - AuditController - ERROR - Creating mysql connection pool failed: 2003: Can't connect to MySQL server on 'localhost:3306' (111 Connection refused) wait mysql ready 2025-06-19T10:28:14.486502Z mysqld_safe Logging to '/data/mysql/log/alert.log'. ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) 2025-06-19T10:28:14.661706Z mysqld_safe Starting mysqld daemon with databases from /data/mysql/data wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) wait mysql ready ERROR 2003 (HY000): Can't connect to MySQL server on '127.1' (111) wait mysql ready 1 1 mysql is ok 2025-06-19 10:29:56,788 - AuditController - INFO - Created a mysql connection pool with size = 3. 2025-06-19 10:29:56,789 - AuditController - INFO - Starting a Label Check Thread to listen k8s' audit label every 10 second. 2025-06-19 10:29:56,789 - AuditController - INFO - Starting a Log Flush Thread to flush audit log every 1 second. delete cluster px-qji `kbcli cluster delete px-qji --auto-approve --namespace ns-smhfl ` Cluster px-qji deleted pod_info:px-qji-cdc-0 2/2 Running 0 4m47s px-qji-cn-0 2/2 Running 0 4m48s px-qji-dn-0 3/3 Terminating 0 12m px-qji-dn-1 3/3 Terminating 0 11m px-qji-dn-2 3/3 Terminating 0 9m51s px-qji-gms-0 3/3 Running 0 12m px-qji-gms-1 3/3 Running 0 11m px-qji-gms-2 3/3 Running 0 10m pod_info:px-qji-dn-0 3/3 Terminating 0 12m px-qji-dn-1 3/3 Terminating 0 11m px-qji-dn-2 3/3 Terminating 0 10m px-qji-gms-0 3/3 Terminating 0 12m px-qji-gms-1 2/3 Terminating 0 11m px-qji-gms-2 2/3 Terminating 0 10m No resources found in ns-smhfl namespace. delete cluster pod done No resources found in ns-smhfl namespace. check cluster resource non-exist OK: pvc No resources found in ns-smhfl namespace. delete cluster done No resources found in ns-smhfl namespace. No resources found in ns-smhfl namespace. No resources found in ns-smhfl namespace. Polardbx Test Suite All Done! --------------------------------------Polardbx (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=polardbx;ClusterVersion=polardbx-v2.3;]|[Description=Create a cluster with the specified cluster definition polardbx and cluster version polardbx-v2.3] [PASSED]|[VerticalScaling]|[ComponentName=cn,cdc]|[Description=VerticalScaling the cluster specify component cn,cdc] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=gms,dn]|[Description=VolumeExpansion the cluster specify component gms,dn] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Restart]|[ComponentName=cn,cdc]|[Description=Restart the cluster specify component cn,cdc] [PASSED]|[HorizontalScaling Out]|[ComponentName=cn,cdc]|[Description=HorizontalScaling Out the cluster specify component cn,cdc] [PASSED]|[HorizontalScaling In]|[ComponentName=cn,cdc]|[Description=HorizontalScaling In the cluster specify component cn,cdc] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]