source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-foqkm ` `kubectl create namespace ns-foqkm` namespace/ns-foqkm created create namespace ns-foqkm done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 47.5M 0 --:--:-- --:--:-- --:--:-- 47.5M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed create apecloud_mysql cluster mysqlo-ufizrh check cluster definition set component name:mysql LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-foqkm namespace. create 3 replica WipeOut apecloud-mysql cluster check cluster version check cluster definition check component definition set component definition by component version no component definitions found `kbcli cluster create mysqlo-ufizrh --termination-policy=WipeOut --cluster-definition=apecloud-mysql --enable-all-logs=false --cluster-version=ac-mysql-8.0.30-1 --set cpu=100m,memory=0.5Gi,replicas=3,storage=1Gi --namespace ns-foqkm ` Cluster mysqlo-ufizrh created check cluster status `kbcli cluster list mysqlo-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysqlo-ufizrh ns-foqkm apecloud-mysql ac-mysql-8.0.30-1 WipeOut Jun 19,2025 18:13 UTC+0800 clusterdefinition.kubeblocks.io/name=apecloud-mysql,clusterversion.kubeblocks.io/name=ac-mysql-8.0.30-1 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysqlo-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysqlo-ufizrh-mysql-0 ns-foqkm mysqlo-ufizrh mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-3-9.us-west-2.compute.internal/172.31.3.9 Jun 19,2025 18:13 UTC+0800 mysqlo-ufizrh-mysql-1 ns-foqkm mysqlo-ufizrh mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-24.us-west-2.compute.internal/172.31.10.24 Jun 19,2025 18:13 UTC+0800 mysqlo-ufizrh-mysql-2 ns-foqkm mysqlo-ufizrh mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-26.us-west-2.compute.internal/172.31.1.26 Jun 19,2025 18:13 UTC+0800 check pod status done check cluster role check cluster role done leader: mysqlo-ufizrh-mysql-1;follower: mysqlo-ufizrh-mysql-0 mysqlo-ufizrh-mysql-2 check cluster connect `echo '' | kbcli cluster connect mysqlo-ufizrh --namespace ns-foqkm` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mysqlo-ufizrh` `kubectl get secrets mysqlo-ufizrh-conn-credential -o jsonpath="***.data.username***"` `kubectl get secrets mysqlo-ufizrh-conn-credential -o jsonpath="***.data.password***"` `kubectl get secrets mysqlo-ufizrh-conn-credential -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:rc2pjmfj;DB_PORT:3306;DB_DATABASE:mysql check pod mysqlo-ufizrh-mysql-1 container_name mysql exist password rc2pjmfj check pod mysqlo-ufizrh-mysql-1 container_name vttablet exist password rc2pjmfj check pod mysqlo-ufizrh-mysql-1 container_name lorry exist password rc2pjmfj check pod mysqlo-ufizrh-mysql-1 container_name config-manager exist password rc2pjmfj No container logs contain secret password. cluster connect Connect to instance mysqlo-ufizrh-mysql-1: out of mysqlo-ufizrh-mysql-1(leader), mysqlo-ufizrh-mysql-0(follower), mysqlo-ufizrh-mysql-2(follower) Defaulted container "mysql" out of: mysql, vttablet, lorry, config-manager, init-data (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `echo 'show variables like "max_connections";' | kbcli cluster connect mysqlo-ufizrh --namespace ns-foqkm ` Defaulted container "mysql" out of: mysql, vttablet, lorry, config-manager, init-data (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Connect to instance mysqlo-ufizrh-mysql-1: out of mysqlo-ufizrh-mysql-1(leader), mysqlo-ufizrh-mysql-0(follower), mysqlo-ufizrh-mysql-2(follower) Variable_name Value max_connections 83 connect cluster Success set max_connections to 83 check cluster definition set component name:orchestrator set component version set component version:orchestrator set service versions:3.2.6 set service versions sorted:3.2.6 no cluster version found set orchestrator component definition set orchestrator component definition orchestrator-raft set replicas first:3,3.2.6 set replicas third:3,3.2.6 set replicas fourth:3,3.2.6 set minimum cmpv service version set minimum cmpv service version replicas:3,3.2.6 REPORT_COUNT:1 CLUSTER_TOPOLOGY:raft topology raft found in cluster definition orchestrator LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-foqkm namespace. termination_policy:Delete create 3 replica Delete orchestrator cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition2: orchestrator-raft by component version:orchestrator apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: orch-ufizrh namespace: ns-foqkm annotations: "kubeblocks.io/extra-env": '***"ORC_TOPOLOGY_PASSWORD":"orchestrator","ORC_META_PASSWORD":"orchestrator","ORC_META_DATABASE":"orchestrator"***' spec: clusterDefinitionRef: orchestrator topology: raft terminationPolicy: Delete componentSpecs: - name: orchestrator serviceVersion: 3.2.6 replicas: 3 resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_orch-ufizrh.yaml` cluster.apps.kubeblocks.io/orch-ufizrh created apply test_create_orch-ufizrh.yaml Success `rm -rf test_create_orch-ufizrh.yaml` check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Creating Jun 19,2025 18:16 UTC+0800 clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-0-64.us-west-2.compute.internal/172.31.0.64 Jun 19,2025 18:16 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=orch-ufizrh` No resources found in ns-foqkm namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:rc2pjmfj;DB_PORT:3306;DB_DATABASE:mysql check pod orch-ufizrh-orchestrator-0 container_name orchestrator exist password rc2pjmfj check pod orch-ufizrh-orchestrator-0 container_name lorry exist password rc2pjmfj No container logs contain secret password. describe cluster `kbcli cluster describe orch-ufizrh --namespace ns-foqkm ` Name: orch-ufizrh Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-foqkm orchestrator Running Delete Endpoints: COMPONENT MODE INTERNAL EXTERNAL orchestrator ReadWrite orch-ufizrh-orchestrator.ns-foqkm.svc.cluster.local:80 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME orchestrator orch-ufizrh-orchestrator-0 secondary Running us-west-2a ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:16 UTC+0800 orchestrator orch-ufizrh-orchestrator-1 primary Running us-west-2a ip-172-31-0-64.us-west-2.compute.internal/172.31.0.64 Jun 19,2025 18:16 UTC+0800 orchestrator orch-ufizrh-orchestrator-2 secondary Running us-west-2a ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS orchestrator false 500m / 500m 512Mi / 512Mi data:1Gi kb-default-sc Images: COMPONENT TYPE IMAGE orchestrator docker.io/apecloud/orchestrator:v3.2.6 Show cluster events: kbcli cluster list-events -n ns-foqkm orch-ufizrh `kbcli cluster label orch-ufizrh app.kubernetes.io/instance- --namespace ns-foqkm ` label "app.kubernetes.io/instance" not found. `kbcli cluster label orch-ufizrh app.kubernetes.io/instance=orch-ufizrh --namespace ns-foqkm ` `kbcli cluster label orch-ufizrh --list --namespace ns-foqkm ` NAME NAMESPACE LABELS orch-ufizrh ns-foqkm app.kubernetes.io/instance=orch-ufizrh clusterdefinition.kubeblocks.io/name=orchestrator clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=orch-ufizrh Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=orch-ufizrh --namespace ns-foqkm ` `kbcli cluster label orch-ufizrh --list --namespace ns-foqkm ` NAME NAMESPACE LABELS orch-ufizrh ns-foqkm app.kubernetes.io/instance=orch-ufizrh case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=orchestrator clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label orch-ufizrh case.name=kbcli.test2 --overwrite --namespace ns-foqkm ` `kbcli cluster label orch-ufizrh --list --namespace ns-foqkm ` NAME NAMESPACE LABELS orch-ufizrh ns-foqkm app.kubernetes.io/instance=orch-ufizrh case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=orchestrator clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label orch-ufizrh case.name- --namespace ns-foqkm ` `kbcli cluster label orch-ufizrh --list --namespace ns-foqkm ` NAME NAMESPACE LABELS orch-ufizrh ns-foqkm app.kubernetes.io/instance=orch-ufizrh clusterdefinition.kubeblocks.io/name=orchestrator clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash ` Defaulted container "orchestrator" out of: orchestrator, lorry, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file ***"Code":"OK","Message":"Application node is healthy","Details":***"Healthy":true,"Hostname":"orch-ufizrh-orchestrator-1","Token":"9b4fb7b880d72ab813b3370f5b1857584ef7731a63e2aedd60ed9920ef1962f3","IsActiveNode":true,"ActiveNode":***"Hostname":"172.31.4.115:10008","Token":"","AppVersion":"","FirstSeenActive":"","LastSeenActive":"","ExtraInfo":"","Command":"","DBBackend":"","LastReported":"0001-01-01T00:00:00Z"***,"Error":null,"AvailableNodes":[***"Hostname":"orch-ufizrh-orchestrator-1","Token":"9b4fb7b880d72ab813b3370f5b1857584ef7731a63e2aedd60ed9920ef1962f3","AppVersion":"3.2.6","FirstSeenActive":"2025-06-19T10:16:54Z","LastSeenActive":"2025-06-19T10:17:50Z","ExtraInfo":"","Command":"","DBBackend":"/data/sqlite/orchestrator.db","LastReported":"0001-01-01T00:00:00Z"***],"RaftLeader":"172.31.4.115:10008","IsRaftLeader":true,"RaftLeaderURI":"http://orch-ufizrh-orchestrator-1.orch-ufizrh-orchestrator-headless:3000","RaftAdvertise":"orch-ufizrh-orchestrator-1.orch-ufizrh-orchestrator-headless","RaftHealthyMembers":["orch-ufizrh-orchestrator-2.orch-ufizrh-orchestrator-headless","orch-ufizrh-orchestrator-0.orch-ufizrh-orchestrator-headless","orch-ufizrh-orchestrator-1.orch-ufizrh-orchestrator-headless"]*** connect cluster Success cluster hscale offline instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: orch-ufizrh-hscaleoffinstance- labels: app.kubernetes.io/instance: orch-ufizrh app.kubernetes.io/managed-by: kubeblocks namespace: ns-foqkm spec: type: HorizontalScaling clusterName: orch-ufizrh force: true horizontalScaling: - componentName: orchestrator scaleIn: onlineInstancesToOffline: - orch-ufizrh-orchestrator-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_orch-ufizrh.yaml` opsrequest.apps.kubeblocks.io/orch-ufizrh-hscaleoffinstance-5jsjd created create test_ops_cluster_orch-ufizrh.yaml Success `rm -rf test_ops_cluster_orch-ufizrh.yaml` check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-hscaleoffinstance-5jsjd ns-foqkm HorizontalScaling orch-ufizrh orchestrator Jun 19,2025 18:17 UTC+0800 ops_status:orch-ufizrh-hscaleoffinstance-5jsjd ns-foqkm HorizontalScaling orch-ufizrh orchestrator Creating -/- Jun 19,2025 18:17 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-0-64.us-west-2.compute.internal/172.31.0.64 Jun 19,2025 18:16 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-1 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-hscaleoffinstance-5jsjd ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:17 UTC+0800 check ops status done ops_status:orch-ufizrh-hscaleoffinstance-5jsjd ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:17 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-hscaleoffinstance-5jsjd --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-hscaleoffinstance-5jsjd patched `kbcli cluster delete-ops --name orch-ufizrh-hscaleoffinstance-5jsjd --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-hscaleoffinstance-5jsjd deleted cluster hscale online instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: orch-ufizrh-hscaleoninstance- labels: app.kubernetes.io/instance: orch-ufizrh app.kubernetes.io/managed-by: kubeblocks namespace: ns-foqkm spec: type: HorizontalScaling clusterName: orch-ufizrh force: true horizontalScaling: - componentName: orchestrator scaleOut: offlineInstancesToOnline: - orch-ufizrh-orchestrator-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_orch-ufizrh.yaml` opsrequest.apps.kubeblocks.io/orch-ufizrh-hscaleoninstance-cblf8 created create test_ops_cluster_orch-ufizrh.yaml Success `rm -rf test_ops_cluster_orch-ufizrh.yaml` check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-hscaleoninstance-cblf8 ns-foqkm HorizontalScaling orch-ufizrh orchestrator Running 0/1 Jun 19,2025 18:18 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-223.us-west-2.compute.internal/172.31.6.223 Jun 19,2025 18:18 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-0-64.us-west-2.compute.internal/172.31.0.64 Jun 19,2025 18:16 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:16 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-hscaleoninstance-cblf8 ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:18 UTC+0800 check ops status done ops_status:orch-ufizrh-hscaleoninstance-cblf8 ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:18 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-hscaleoninstance-cblf8 --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-hscaleoninstance-cblf8 patched `kbcli cluster delete-ops --name orch-ufizrh-hscaleoninstance-cblf8 --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-hscaleoninstance-cblf8 deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop orch-ufizrh --auto-approve --force=true --namespace ns-foqkm ` OpsRequest orch-ufizrh-stop-tfc9m created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-stop-tfc9m -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-stop-tfc9m ns-foqkm Stop orch-ufizrh orchestrator Running 0/3 Jun 19,2025 18:19 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Stopping Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-stop-tfc9m ns-foqkm Stop orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:19 UTC+0800 check ops status done ops_status:orch-ufizrh-stop-tfc9m ns-foqkm Stop orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-stop-tfc9m --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-stop-tfc9m patched `kbcli cluster delete-ops --name orch-ufizrh-stop-tfc9m --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-stop-tfc9m deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start orch-ufizrh --force=true --namespace ns-foqkm ` OpsRequest orch-ufizrh-start-mcntl created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-start-mcntl -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-start-mcntl ns-foqkm Start orch-ufizrh orchestrator Running 0/3 Jun 19,2025 18:21 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Failed Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-15-168.us-west-2.compute.internal/172.31.15.168 Jun 19,2025 18:21 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:21 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-6-223.us-west-2.compute.internal/172.31.6.223 Jun 19,2025 18:21 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-start-mcntl ns-foqkm Start orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:21 UTC+0800 check ops status done ops_status:orch-ufizrh-start-mcntl ns-foqkm Start orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-start-mcntl --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-start-mcntl patched `kbcli cluster delete-ops --name orch-ufizrh-start-mcntl --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-start-mcntl deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in orch-ufizrh namespace. `kbcli cluster hscale orch-ufizrh --auto-approve --force=true --components orchestrator --replicas 4 --namespace ns-foqkm ` OpsRequest orch-ufizrh-horizontalscaling-xpltz created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-horizontalscaling-xpltz -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-xpltz ns-foqkm HorizontalScaling orch-ufizrh orchestrator Creating -/- Jun 19,2025 18:21 UTC+0800 cluster restart `kbcli cluster restart orch-ufizrh --auto-approve --force=true --namespace ns-foqkm ` OpsRequest orch-ufizrh-restart-5jfzg created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-restart-5jfzg -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-xpltz ns-foqkm HorizontalScaling orch-ufizrh orchestrator Running 0/1 Jun 19,2025 18:21 UTC+0800 orch-ufizrh-restart-5jfzg ns-foqkm Restart orch-ufizrh orchestrator Running 0/4 Jun 19,2025 18:22 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Failed cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Failed cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-0-64.us-west-2.compute.internal/172.31.0.64 Jun 19,2025 18:24 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:23 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:25 UTC+0800 orch-ufizrh-orchestrator-3 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-xpltz ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:21 UTC+0800 orch-ufizrh-restart-5jfzg ns-foqkm Restart orch-ufizrh orchestrator Succeed 4/4 Jun 19,2025 18:22 UTC+0800 check ops status done ops_status:orch-ufizrh-restart-5jfzg ns-foqkm Restart orch-ufizrh orchestrator Succeed 4/4 Jun 19,2025 18:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-restart-5jfzg --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-restart-5jfzg patched `kbcli cluster delete-ops --name orch-ufizrh-restart-5jfzg --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-restart-5jfzg deleted check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-0-64.us-west-2.compute.internal/172.31.0.64 Jun 19,2025 18:24 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-4.us-west-2.compute.internal/172.31.5.4 Jun 19,2025 18:23 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:25 UTC+0800 orch-ufizrh-orchestrator-3 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:22 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done No resources found in orch-ufizrh namespace. check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-xpltz ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:21 UTC+0800 check ops status done ops_status:orch-ufizrh-horizontalscaling-xpltz ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-horizontalscaling-xpltz --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-horizontalscaling-xpltz patched `kbcli cluster delete-ops --name orch-ufizrh-horizontalscaling-xpltz --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-horizontalscaling-xpltz deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in orch-ufizrh namespace. `kbcli cluster hscale orch-ufizrh --auto-approve --force=true --components orchestrator --replicas 3 --namespace ns-foqkm ` OpsRequest orch-ufizrh-horizontalscaling-5x4br created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-horizontalscaling-5x4br -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-5x4br ns-foqkm HorizontalScaling orch-ufizrh orchestrator Running 0/1 Jun 19,2025 18:26 UTC+0800 cluster restart `kbcli cluster restart orch-ufizrh --auto-approve --force=true --namespace ns-foqkm ` OpsRequest orch-ufizrh-restart-nnwxt created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-restart-nnwxt -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-5x4br ns-foqkm HorizontalScaling orch-ufizrh orchestrator Running 0/1 Jun 19,2025 18:26 UTC+0800 orch-ufizrh-restart-nnwxt ns-foqkm Restart orch-ufizrh orchestrator Running 0/3 Jun 19,2025 18:26 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:28 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:27 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-5x4br ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:26 UTC+0800 orch-ufizrh-restart-nnwxt ns-foqkm Restart orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:26 UTC+0800 check ops status done ops_status:orch-ufizrh-restart-nnwxt ns-foqkm Restart orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:26 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-restart-nnwxt --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-restart-nnwxt patched `kbcli cluster delete-ops --name orch-ufizrh-restart-nnwxt --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-restart-nnwxt deleted check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:28 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-1-27.us-west-2.compute.internal/172.31.1.27 Jun 19,2025 18:27 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:28 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done No resources found in orch-ufizrh namespace. check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-horizontalscaling-5x4br ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:26 UTC+0800 check ops status done ops_status:orch-ufizrh-horizontalscaling-5x4br ns-foqkm HorizontalScaling orch-ufizrh orchestrator Succeed 1/1 Jun 19,2025 18:26 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-horizontalscaling-5x4br --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-horizontalscaling-5x4br patched `kbcli cluster delete-ops --name orch-ufizrh-horizontalscaling-5x4br --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-horizontalscaling-5x4br deleted cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart orch-ufizrh --auto-approve --force=true --namespace ns-foqkm ` OpsRequest orch-ufizrh-restart-bgkhz created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-restart-bgkhz -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-restart-bgkhz ns-foqkm Restart orch-ufizrh orchestrator Running 0/3 Jun 19,2025 18:29 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:30 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-restart-bgkhz ns-foqkm Restart orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:29 UTC+0800 check ops status done ops_status:orch-ufizrh-restart-bgkhz ns-foqkm Restart orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-restart-bgkhz --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-restart-bgkhz patched `kbcli cluster delete-ops --name orch-ufizrh-restart-bgkhz --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-restart-bgkhz deleted cluster update monitor KUBEBLOCKS NAMESPACE:kb-zkhtd get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster orch-ufizrh --namespace ns-foqkm --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/orch-ufizrh patched check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:30 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done create curl-jq pod orch-ufizrh-kb-curl `kubectl run --namespace kb-zkhtd orch-ufizrh-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/orch-ufizrh-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE orch-ufizrh-kb-curl 0/1 ContainerCreating 0 1s check pod orch-ufizrh-kb-curl status done `kubectl wait --for=condition=Ready pod orch-ufizrh-kb-curl --namespace kb-zkhtd` pod/orch-ufizrh-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE orch-ufizrh-kb-curl 1/1 Running 0 5s kubectl exec -it orch-ufizrh-kb-curl --namespace kb-zkhtd -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it orch-ufizrh-kb-curl --namespace kb-zkhtd -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 121 100 121 0 0 32223 0 --:--:-- --:--:-- --:--:-- 40333 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750329180.730,"9.899999999999999"]***]*** k8s_pod_cpu_limit: 1750329180.730 monitor cluster Success `kubectl delete pod orch-ufizrh-kb-curl --namespace kb-zkhtd --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "orch-ufizrh-kb-curl" force deleted check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:30 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done 6 cluster update enable-all-logs `kbcli cluster update orch-ufizrh --enable-all-logs=true --namespace ns-foqkm ` error: failed to reconfigure log variables of target cluster: no config templates for component orchestrator check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:30 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done `kubectl get pvc -l app.kubernetes.io/instance=orch-ufizrh,apps.kubeblocks.io/component-name=orchestrator,apps.kubeblocks.io/vct-name=data --namespace ns-foqkm ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in orch-ufizrh namespace. `kbcli cluster volume-expand orch-ufizrh --auto-approve --force=true --components orchestrator --volume-claim-templates data --storage 3Gi --namespace ns-foqkm ` OpsRequest orch-ufizrh-volumeexpansion-8fqpk created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-volumeexpansion-8fqpk -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-volumeexpansion-8fqpk ns-foqkm VolumeExpansion orch-ufizrh orchestrator Running 0/3 Jun 19,2025 18:33 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-14-103.us-west-2.compute.internal/172.31.14.103 Jun 19,2025 18:31 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:3Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:30 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done No resources found in orch-ufizrh namespace. check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-volumeexpansion-8fqpk ns-foqkm VolumeExpansion orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:33 UTC+0800 check ops status done ops_status:orch-ufizrh-volumeexpansion-8fqpk ns-foqkm VolumeExpansion orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-volumeexpansion-8fqpk --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-volumeexpansion-8fqpk patched `kbcli cluster delete-ops --name orch-ufizrh-volumeexpansion-8fqpk --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-volumeexpansion-8fqpk deleted cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale orch-ufizrh --auto-approve --force=true --components orchestrator --cpu 600m --memory 0.6Gi --namespace ns-foqkm ` OpsRequest orch-ufizrh-verticalscaling-f8bqf created successfully, you can view the progress: kbcli cluster describe-ops orch-ufizrh-verticalscaling-f8bqf -n ns-foqkm check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-verticalscaling-f8bqf ns-foqkm VerticalScaling orch-ufizrh orchestrator Running 0/3 Jun 19,2025 18:34 UTC+0800 check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:36 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:36 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-223.us-west-2.compute.internal/172.31.6.223 Jun 19,2025 18:35 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done check ops status `kbcli cluster list-ops orch-ufizrh --status all --namespace ns-foqkm ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME orch-ufizrh-verticalscaling-f8bqf ns-foqkm VerticalScaling orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:34 UTC+0800 check ops status done ops_status:orch-ufizrh-verticalscaling-f8bqf ns-foqkm VerticalScaling orch-ufizrh orchestrator Succeed 3/3 Jun 19,2025 18:34 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests orch-ufizrh-verticalscaling-f8bqf --namespace ns-foqkm ` opsrequest.apps.kubeblocks.io/orch-ufizrh-verticalscaling-f8bqf patched `kbcli cluster delete-ops --name orch-ufizrh-verticalscaling-f8bqf --force --auto-approve --namespace ns-foqkm ` OpsRequest orch-ufizrh-verticalscaling-f8bqf deleted cluster update terminationPolicy WipeOut `kbcli cluster update orch-ufizrh --termination-policy=WipeOut --namespace ns-foqkm ` cluster.apps.kubeblocks.io/orch-ufizrh updated check cluster status `kbcli cluster list orch-ufizrh --show-labels --namespace ns-foqkm ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS orch-ufizrh ns-foqkm orchestrator WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=orch-ufizrh,clusterdefinition.kubeblocks.io/name=orchestrator,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orch-ufizrh --namespace ns-foqkm ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orch-ufizrh-orchestrator-0 ns-foqkm orch-ufizrh orchestrator Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-13-149.us-west-2.compute.internal/172.31.13.149 Jun 19,2025 18:36 UTC+0800 orch-ufizrh-orchestrator-1 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-3-103.us-west-2.compute.internal/172.31.3.103 Jun 19,2025 18:36 UTC+0800 orch-ufizrh-orchestrator-2 ns-foqkm orch-ufizrh orchestrator Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:3Gi ip-172-31-6-223.us-west-2.compute.internal/172.31.6.223 Jun 19,2025 18:35 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orch-ufizrh-orchestrator:80/api/health ' | kubectl exec -it orch-ufizrh-orchestrator-0 --namespace ns-foqkm -- bash` check cluster connect done cluster list-logs `kbcli cluster list-logs orch-ufizrh --namespace ns-foqkm ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update orch-ufizrh --enable-all-logs=true --namespace ns-foqkm Error from server (NotFound): pods "orch-ufizrh-orchestrator-0" not found cluster logs `kbcli cluster logs orch-ufizrh --tail 30 --namespace ns-foqkm ` Defaulted container "orchestrator" out of: orchestrator, lorry, init-lorry (init) 2025/06/19 10:37:31 [ERR] raft: Failed to heartbeat to 172.31.5.241:10008: dial tcp 172.31.5.241:10008: connect: no route to host 2025/06/19 10:37:31 [ERR] raft: Failed to AppendEntries to 172.31.5.241:10008: dial tcp 172.31.5.241:10008: connect: no route to host 2025/06/19 10:37:31 [ERR] raft: Failed to heartbeat to 172.31.4.115:10008: dial tcp 172.31.4.115:10008: connect: no route to host 2025/06/19 10:37:31 [ERR] raft: Failed to AppendEntries to 172.31.4.115:10008: dial tcp 172.31.4.115:10008: connect: no route to host 2025/06/19 10:37:31 [ERR] raft: Failed to AppendEntries to 172.31.12.185:10008: dial tcp 172.31.12.185:10008: connect: no route to host 2025/06/19 10:37:31 [ERR] raft: Failed to heartbeat to 172.31.12.185:10008: dial tcp 172.31.12.185:10008: connect: no route to host 2025/06/19 10:37:32 [ERR] raft: Failed to AppendEntries to 172.31.10.125:10008: dial tcp 172.31.10.125:10008: i/o timeout 2025/06/19 10:37:32 [ERR] raft: Failed to AppendEntries to 172.31.3.59:10008: dial tcp 172.31.3.59:10008: i/o timeout 2025/06/19 10:37:32 [ERR] raft: Failed to AppendEntries to 172.31.8.32:10008: dial tcp 172.31.8.32:10008: i/o timeout 2025/06/19 10:37:32 [ERR] raft: Failed to AppendEntries to 172.31.11.110:10008: dial tcp 172.31.11.110:10008: i/o timeout 2025/06/19 10:37:32 [ERR] raft: Failed to AppendEntries to 172.31.9.159:10008: dial tcp 172.31.9.159:10008: i/o timeout 2025/06/19 10:37:32 [ERR] raft: Failed to AppendEntries to 172.31.9.215:10008: dial tcp 172.31.9.215:10008: i/o timeout 2025/06/19 10:37:33 [ERR] raft: Failed to heartbeat to 172.31.8.32:10008: dial tcp 172.31.8.32:10008: i/o timeout 2025/06/19 10:37:33 [ERR] raft: Failed to heartbeat to 172.31.10.125:10008: dial tcp 172.31.10.125:10008: i/o timeout 2025/06/19 10:37:33 [ERR] raft: Failed to heartbeat to 172.31.3.59:10008: dial tcp 172.31.3.59:10008: i/o timeout 2025/06/19 10:37:33 [ERR] raft: Failed to heartbeat to 172.31.11.110:10008: dial tcp 172.31.11.110:10008: i/o timeout 2025/06/19 10:37:33 [ERR] raft: Failed to heartbeat to 172.31.9.215:10008: dial tcp 172.31.9.215:10008: i/o timeout 2025/06/19 10:37:33 [ERR] raft: Failed to heartbeat to 172.31.9.159:10008: dial tcp 172.31.9.159:10008: i/o timeout [martini] Started GET /api/leader-check for 127.0.0.1:48958 [martini] Completed 200 OK in 561.609µs [martini] Started GET /api/raft-follower-health-report/e259356a/orch-ufizrh-orchestrator-0.orch-ufizrh-orchestrator-headless/orch-ufizrh-orchestrator-0.orch-ufizrh-orchestrator-headless for 172.31.9.128:35090 [martini] Completed 200 OK in 1.407705ms [martini] Started GET /api/raft-follower-health-report/e259356a/orch-ufizrh-orchestrator-1.orch-ufizrh-orchestrator-headless/orch-ufizrh-orchestrator-1.orch-ufizrh-orchestrator-headless for 172.31.3.244:47296 [martini] Started GET /api/raft-follower-health-report/e259356a/orch-ufizrh-orchestrator-2.orch-ufizrh-orchestrator-headless/orch-ufizrh-orchestrator-2.orch-ufizrh-orchestrator-headless for 172.31.3.114:37518 [martini] Completed 200 OK in 1.382714ms [martini] Completed 200 OK in 1.24523ms [martini] Started GET /api/health for 172.31.13.149:52980 [martini] Completed 200 OK in 4.111332ms [martini] Started GET /api/health for 172.31.13.149:19114 [martini] Completed 200 OK in 2.540782ms delete cluster orch-ufizrh `kbcli cluster delete orch-ufizrh --auto-approve --namespace ns-foqkm ` Cluster orch-ufizrh deleted pod_info:orch-ufizrh-orchestrator-0 2/2 Terminating 0 49s orch-ufizrh-orchestrator-1 2/2 Terminating 0 100s orch-ufizrh-orchestrator-2 2/2 Terminating 0 2m35s pod_info:orch-ufizrh-orchestrator-0 2/2 Terminating 0 70s orch-ufizrh-orchestrator-1 2/2 Terminating 0 2m1s orch-ufizrh-orchestrator-2 2/2 Terminating 0 2m56s No resources found in ns-foqkm namespace. delete cluster pod done No resources found in ns-foqkm namespace. check cluster resource non-exist OK: pvc No resources found in ns-foqkm namespace. delete cluster done No resources found in ns-foqkm namespace. No resources found in ns-foqkm namespace. No resources found in ns-foqkm namespace. Orchestrator Cluster Test Suite All Done! delete cluster mysqlo-ufizrh `kbcli cluster delete mysqlo-ufizrh --auto-approve --namespace ns-foqkm ` Cluster mysqlo-ufizrh deleted pod_info:mysqlo-ufizrh-mysql-0 4/4 Running 0 25m mysqlo-ufizrh-mysql-1 4/4 Running 0 25m mysqlo-ufizrh-mysql-2 4/4 Running 0 25m pod_info:mysqlo-ufizrh-mysql-0 3/4 Terminating 0 25m mysqlo-ufizrh-mysql-1 3/4 Terminating 0 25m mysqlo-ufizrh-mysql-2 3/4 Terminating 0 25m No resources found in ns-foqkm namespace. delete cluster pod done No resources found in ns-foqkm namespace. check cluster resource non-exist OK: pvc No resources found in ns-foqkm namespace. delete cluster done No resources found in ns-foqkm namespace. No resources found in ns-foqkm namespace. No resources found in ns-foqkm namespace. [PASSED]|[Create]|[ClusterDefinition=apecloud-mysql;ClusterVersion=ac-mysql-8.0.30-1;]|[Description=Create a cluster with the specified cluster definition apecloud-mysql and cluster version ac-mysql-8.0.30-1] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] --------------------------------------Orchestrator Cluster (Topology = raft Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=raft;ComponentVersion=orchestrator;ServiceVersion=3.2.6;]|[Description=Create a cluster with the specified topology raft and component version orchestrator and service version 3.2.6] [PASSED]|[Connect]|[ComponentName=orchestrator]|[Description=Connect to the cluster] [PASSED]|[HscaleOfflineInstances]|[ComponentName=orchestrator]|[Description=Hscale the cluster instances offline specify component orchestrator] [PASSED]|[HscaleOnlineInstances]|[ComponentName=orchestrator]|[Description=Hscale the cluster instances online specify component orchestrator] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[HorizontalScaling Out]|[ComponentName=orchestrator]|[Description=HorizontalScaling Out the cluster specify component orchestrator] [PASSED]|[HorizontalScaling In]|[ComponentName=orchestrator]|[Description=HorizontalScaling In the cluster specify component orchestrator] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[VolumeExpansion]|[ComponentName=orchestrator]|[Description=VolumeExpansion the cluster specify component orchestrator] [PASSED]|[VerticalScaling]|[ComponentName=orchestrator]|[Description=VerticalScaling the cluster specify component orchestrator] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]