source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-yahzs ` `kubectl create namespace ns-yahzs` namespace/ns-yahzs created create namespace ns-yahzs done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.0` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 110M 0 --:--:-- --:--:-- --:--:-- 110M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.0 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Kubernetes Env: v1.32.5-eks-5d4a308 check snapshot controller check snapshot controller done eks default-vsc found POD_RESOURCES: No resources found found default storage class: gp3 KubeBlocks version is:1.0.0 skip upgrade KubeBlocks current KubeBlocks version: 1.0.0 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed create orchestrator cluster orcm-qdhftk check component definition set component name:orchestrator set orchestrator component definition set orchestrator component definition orchestrator-raft-1.0.0-alpha.0 LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-yahzs namespace. create 3 replica WipeOut orchestrator cluster check component definition set component definition by component version no component definitions found apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: orcm-qdhftk namespace: ns-yahzs spec: clusterDef: orchestrator topology: raft terminationPolicy: WipeOut services: - name: orchestrator componentSelector: orchestrator spec: ports: - name: orc-http port: 80 componentSpecs: - name: orchestrator serviceVersion: 3.2.6 replicas: 3 resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_orcm-qdhftk.yaml` cluster.apps.kubeblocks.io/orcm-qdhftk created apply test_create_orcm-qdhftk.yaml Success `rm -rf test_create_orcm-qdhftk.yaml` check cluster status `kbcli cluster list orcm-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS orcm-qdhftk ns-yahzs orchestrator WipeOut Creating May 28,2025 11:35 UTC+0800 clusterdefinition.kubeblocks.io/name=orchestrator cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances orcm-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME orcm-qdhftk-orchestrator-0 ns-yahzs orcm-qdhftk orchestrator Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:35 UTC+0800 orcm-qdhftk-orchestrator-1 ns-yahzs orcm-qdhftk orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-13-207.us-west-2.compute.internal/172.31.13.207 May 28,2025 11:36 UTC+0800 orcm-qdhftk-orchestrator-2 ns-yahzs orcm-qdhftk orchestrator Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-188.us-west-2.compute.internal/172.31.4.188 May 28,2025 11:36 UTC+0800 check pod status done check cluster connect ` echo 'curl -s http://orcm-qdhftk-orchestrator:80/api/health ' | kubectl exec -it orcm-qdhftk-orchestrator-0 --namespace ns-yahzs -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=orcm-qdhftk` set secret: orcm-qdhftk-orchestrator-account-orchestrator `kubectl get secrets orcm-qdhftk-orchestrator-account-orchestrator -o jsonpath="***.data.username***"` `kubectl get secrets orcm-qdhftk-orchestrator-account-orchestrator -o jsonpath="***.data.password***"` `kubectl get secrets orcm-qdhftk-orchestrator-account-orchestrator -o jsonpath="***.data.port***"` DB_USERNAME:orchestrator;DB_PASSWORD:7Q7c858z0XK6Vc6i;DB_PORT:;DB_DATABASE: check pod orcm-qdhftk-orchestrator-0 container_name orchestrator exist password 7Q7c858z0XK6Vc6i check pod orcm-qdhftk-orchestrator-0 container_name kbagent exist password 7Q7c858z0XK6Vc6i No container logs contain secret password. check component definition set component name:mysql set component version set component version:mysql set service versions:8.4.2,8.4.1,8.4.0,8.0.39,8.0.38,8.0.37,8.0.36,8.0.35,8.0.34,8.0.33,8.0.32,8.0.31,8.0.30,5.7.44 set service versions sorted:5.7.44,8.0.30,8.0.31,8.0.32,8.0.33,8.0.34,8.0.35,8.0.36,8.0.37,8.0.38,8.0.39,8.4.0,8.4.1,8.4.2 set mysql component definition set mysql component definition set replicas first:2,5.7.44|2,8.0.30|2,8.0.31|2,8.0.32|2,8.0.33|2,8.0.34|2,8.0.35|2,8.0.36|2,8.0.37|2,8.0.38|2,8.0.39 set replicas third:2,8.0.30 set replicas fourth:2,8.0.30 set minimum cmpv service version set minimum cmpv service version replicas:2,8.0.30 REPORT_COUNT:1 CLUSTER_TOPOLOGY:orc topology orc found in cluster definition mysql set mysql component definition set mysql component definition LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 10 No resources found in ns-yahzs namespace. termination_policy:Delete create 2 replica Delete mysql cluster check component definition set component definition by component version check cmpd by labels set component definition1: mysql-8.0-1.0.0-alpha.0 by component version:mysql apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: mysql-qdhftk namespace: ns-yahzs spec: clusterDef: mysql topology: orc terminationPolicy: Delete componentSpecs: - name: mysql serviceVersion: 8.0.30 disableExporter: true replicas: 2 resources: limits: cpu: 500m memory: 0.5Gi requests: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi serviceRefs: - name: orchestrator namespace: ns-yahzs clusterServiceSelector: cluster: orcm-qdhftk service: component: orchestrator service: orchestrator port: orc-http credential: component: orchestrator name: orchestrator `kubectl apply -f test_create_mysql-qdhftk.yaml` cluster.apps.kubeblocks.io/mysql-qdhftk created apply test_create_mysql-qdhftk.yaml Success `rm -rf test_create_mysql-qdhftk.yaml` check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Creating May 28,2025 11:37 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check pod mysql-qdhftk-mysql-0 container_name mysql exist password Ab316X41iY check pod mysql-qdhftk-mysql-0 container_name kbagent exist password Ab316X41iY check pod mysql-qdhftk-mysql-0 container_name config-manager exist password Ab316X41iY No container logs contain secret password. describe cluster `kbcli cluster describe mysql-qdhftk --namespace ns-yahzs ` Name: mysql-qdhftk Created Time: May 28,2025 11:37 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-yahzs mysql orc Running Delete Endpoints: COMPONENT INTERNAL EXTERNAL mysql mysql-qdhftk-mysql-0.ns-yahzs.svc.cluster.local:3306 mysql-qdhftk-mysql-1.ns-yahzs.svc.cluster.local:3306 mysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local:3306 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql 8.0.30 mysql-qdhftk-mysql-0 primary Running us-west-2a ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql 8.0.30 mysql-qdhftk-mysql-1 secondary Running us-west-2a ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql 500m / 500m 512Mi / 512Mi data:10Gi kb-default-sc Images: COMPONENT COMPONENT-DEFINITION IMAGE mysql mysql-orc-8.0-1.0.0-alpha.0 docker.io/apecloud/mysql:8.0.30 docker.io/apecloud/kubeblocks-tools:1.0.0 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-yahzs mysql-qdhftk `kbcli cluster label mysql-qdhftk app.kubernetes.io/instance- --namespace ns-yahzs ` label "app.kubernetes.io/instance" not found. `kbcli cluster label mysql-qdhftk app.kubernetes.io/instance=mysql-qdhftk --namespace ns-yahzs ` `kbcli cluster label mysql-qdhftk --list --namespace ns-yahzs ` NAME NAMESPACE LABELS mysql-qdhftk ns-yahzs app.kubernetes.io/instance=mysql-qdhftk clusterdefinition.kubeblocks.io/name=mysql label cluster app.kubernetes.io/instance=mysql-qdhftk Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=mysql-qdhftk --namespace ns-yahzs ` `kbcli cluster label mysql-qdhftk --list --namespace ns-yahzs ` NAME NAMESPACE LABELS mysql-qdhftk ns-yahzs app.kubernetes.io/instance=mysql-qdhftk case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=mysql label cluster case.name=kbcli.test1 Success `kbcli cluster label mysql-qdhftk case.name=kbcli.test2 --overwrite --namespace ns-yahzs ` `kbcli cluster label mysql-qdhftk --list --namespace ns-yahzs ` NAME NAMESPACE LABELS mysql-qdhftk ns-yahzs app.kubernetes.io/instance=mysql-qdhftk case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=mysql label cluster case.name=kbcli.test2 Success `kbcli cluster label mysql-qdhftk case.name- --namespace ns-yahzs ` `kbcli cluster label mysql-qdhftk --list --namespace ns-yahzs ` NAME NAMESPACE LABELS mysql-qdhftk ns-yahzs app.kubernetes.io/instance=mysql-qdhftk clusterdefinition.kubeblocks.io/name=mysql delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: mysql-qdhftk-mysql-0.mysql-qdhftk-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 1 Master_Log_File: mysql-qdhftk-mysql-0-bin.000003 Read_Master_Log_Pos: 3512 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 3758 Relay_Master_Log_File: mysql-qdhftk-mysql-0-bin.000003 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 3512 Relay_Log_Space: 3962 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 38cf5c64-3b75-11f0-bf1d-6e9affc73c51 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 38cf5c64-3b75-11f0-bf1d-6e9affc73c51:1-10 Executed_Gtid_Set: 38cf5c64-3b75-11f0-bf1d-6e9affc73c51:1-10 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 1 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pAb316X41iY -P3306 " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 83 connect cluster Success set max_connections to 83 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-mysql-qdhftk" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mysql-qdhftk --namespace ns-yahzs ` Error from server (NotFound): pods "test-db-client-executionloop-mysql-qdhftk" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-mysql-qdhftk" not found `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-mysql-qdhftk namespace: ns-yahzs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local" - "--user" - "root" - "--password" - "Ab316X41iY" - "--port" - "3306" - "--dbtype" - "mysql" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-mysql-qdhftk.yaml` pod/test-db-client-executionloop-mysql-qdhftk created apply test-db-client-executionloop-mysql-qdhftk.yaml Success `rm -rf test-db-client-executionloop-mysql-qdhftk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 59s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 65s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 1/1 Running 0 70s check pod test-db-client-executionloop-mysql-qdhftk status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-qdhftk 0/1 Completed 0 76s check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done Execution loop start: create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; java.io.IOException: Failed to execute query: java.sql.SQLException: The MySQL server is running with the --super-read-only option so it cannot execute this statement [2025-05-28 03:41:09.436] Connection error occurred! create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; java.io.IOException: Failed to execute query: java.sql.SQLException: The MySQL server is running with the --super-read-only option so it cannot execute this statement [ 1s ] executions total: 2 successful: 0 failed: 2 disconnect: 1 create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; java.io.IOException: Failed to execute query: java.sql.SQLException: The MySQL server is running with the --super-read-only option so it cannot execute this statement [ 2s ] executions total: 3 successful: 0 failed: 3 disconnect: 1 create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255), tinyint_col TINYINT, smallint_col SMALLINT, mediumint_col MEDIUMINT, int_col INT, bigint_col BIGINT, float_col FLOAT, double_col DOUBLE, decimal_col DECIMAL(10, 2), date_col DATE, time_col TIME, datetime_col DATETIME, timestamp_col TIMESTAMP, year_col YEAR, char_col CHAR(10), text_col TEXT, blob_col BLOB, binary_col BINARY(10), varbinary_col VARBINARY(255), enum_col ENUM('Option1', 'Option2', 'Option3'), set_col SET('Value1', 'Value2', 'Value3') ); Execution loop start:INSERT INTO executions_loop.executions_loop_table (value, tinyint_col, smallint_col, mediumint_col, int_col, bigint_col, float_col, double_col, decimal_col, date_col, time_col, datetime_col, timestamp_col, year_col, char_col, text_col, blob_col, binary_col, varbinary_col, enum_col, set_col) VALUES ('executions_loop_test_1', 95, 25005, 4744958, -1001700054, 8820793998299220721, 0.38289326, 0.43314803618442144, 50.153322286260604, '2025-05-28', '03:41:12', '2025-05-28 03:41:12.634', CURRENT_TIMESTAMP, 1987, 'O8TUZ2HK8j', 'sYaenXE9Ek6iOWwiPTUOdQH5CmQLU3h8U2pUypw8WCEmeW3iPXCS8wvl7i42H8IrHMwu8yubFENVrv6t5ux6FuI4rwrfce2SbvvmG41zBNclNRXyxzaoPvIg8nBa3kCOsn9jOIqpaNDoJp5kNB19RHUp2S9bXEMxdY6ibgduoSWuCwFOYMQ2vIbiz5oKqDPWwe0R2gTDStSDaG7dI0Lzy1JhJ4spD6mevHa9uhtRwddkAMqlh4EBh7tgbglAF3p', UNHEX(REPLACE('e7c756725155e39cba12', ' ', '')), UNHEX(REPLACE('60c423cb7ac4631c749d', ' ', '')), UNHEX(REPLACE('45e56396054ff16e073652982b3deef870b01cd84d5e52cc37d435a32f3cef30a5d78ab8c8ea87b6bb14cfbb87a830e36d21fb5cf96726ecc8388cfce688527e02c4ad575d17a932c23d89355111c3132227b724ac07b9f737a5d1ab04fe55cd256f4ab95e444edb0e48e2ff96383c825225b9e470f935b3ffa898225b2bee9f6d91549b678ff0cf467aa0800967dd67949812c1940387a8924a47309d7ee8a84a35a1a59a002737438da9af7f3dd6d5d4dcb06a2d7b499fd571abd0e49bcb46c365efe01cd71f98963e63b0731502d5d243b2bd90d83f379f22c2df2a463afcf12ce82b95e5464801b3a33e83f1c08ea17bd7c3da5bb1523d873bf8d2d79a', ' ', '')), 'Option2', 'Value1' ); [2025-05-28 03:41:09.436] Connection error occurred! [2025-05-28 03:41:12.736] Connection successfully recovered! The connection was restored in 3300 milliseconds. [ 3s ] executions total: 4 successful: 1 failed: 3 disconnect: 1 [ 4s ] executions total: 81 successful: 78 failed: 3 disconnect: 1 [ 5s ] executions total: 159 successful: 156 failed: 3 disconnect: 1 [ 6s ] executions total: 241 successful: 238 failed: 3 disconnect: 1 [ 7s ] executions total: 329 successful: 326 failed: 3 disconnect: 1 [ 8s ] executions total: 426 successful: 423 failed: 3 disconnect: 1 [ 9s ] executions total: 520 successful: 517 failed: 3 disconnect: 1 [ 10s ] executions total: 618 successful: 615 failed: 3 disconnect: 1 [ 11s ] executions total: 703 successful: 700 failed: 3 disconnect: 1 [ 12s ] executions total: 796 successful: 793 failed: 3 disconnect: 1 [ 13s ] executions total: 895 successful: 892 failed: 3 disconnect: 1 [ 14s ] executions total: 987 successful: 984 failed: 3 disconnect: 1 [ 15s ] executions total: 1077 successful: 1074 failed: 3 disconnect: 1 [ 16s ] executions total: 1178 successful: 1175 failed: 3 disconnect: 1 [ 17s ] executions total: 1278 successful: 1275 failed: 3 disconnect: 1 [ 18s ] executions total: 1392 successful: 1389 failed: 3 disconnect: 1 [ 19s ] executions total: 1491 successful: 1488 failed: 3 disconnect: 1 [ 20s ] executions total: 1582 successful: 1579 failed: 3 disconnect: 1 [ 21s ] executions total: 1674 successful: 1671 failed: 3 disconnect: 1 [ 22s ] executions total: 1772 successful: 1769 failed: 3 disconnect: 1 [ 23s ] executions total: 1874 successful: 1871 failed: 3 disconnect: 1 [ 24s ] executions total: 1980 successful: 1977 failed: 3 disconnect: 1 [ 25s ] executions total: 2076 successful: 2073 failed: 3 disconnect: 1 [ 26s ] executions total: 2173 successful: 2170 failed: 3 disconnect: 1 [ 27s ] executions total: 2271 successful: 2268 failed: 3 disconnect: 1 [ 28s ] executions total: 2376 successful: 2373 failed: 3 disconnect: 1 [ 29s ] executions total: 2468 successful: 2465 failed: 3 disconnect: 1 [ 30s ] executions total: 2566 successful: 2563 failed: 3 disconnect: 1 [ 31s ] executions total: 2675 successful: 2672 failed: 3 disconnect: 1 [ 32s ] executions total: 2785 successful: 2782 failed: 3 disconnect: 1 [ 33s ] executions total: 2890 successful: 2887 failed: 3 disconnect: 1 [ 34s ] executions total: 3000 successful: 2997 failed: 3 disconnect: 1 [ 35s ] executions total: 3086 successful: 3083 failed: 3 disconnect: 1 [ 36s ] executions total: 3192 successful: 3189 failed: 3 disconnect: 1 [ 37s ] executions total: 3279 successful: 3276 failed: 3 disconnect: 1 [ 38s ] executions total: 3387 successful: 3384 failed: 3 disconnect: 1 [ 39s ] executions total: 3493 successful: 3490 failed: 3 disconnect: 1 [ 40s ] executions total: 3588 successful: 3585 failed: 3 disconnect: 1 [ 41s ] executions total: 3683 successful: 3680 failed: 3 disconnect: 1 [ 42s ] executions total: 3783 successful: 3780 failed: 3 disconnect: 1 [ 43s ] executions total: 3885 successful: 3882 failed: 3 disconnect: 1 [ 44s ] executions total: 3986 successful: 3983 failed: 3 disconnect: 1 [ 45s ] executions total: 4090 successful: 4087 failed: 3 disconnect: 1 [ 46s ] executions total: 4181 successful: 4178 failed: 3 disconnect: 1 [ 47s ] executions total: 4281 successful: 4278 failed: 3 disconnect: 1 [ 48s ] executions total: 4391 successful: 4388 failed: 3 disconnect: 1 [ 49s ] executions total: 4495 successful: 4492 failed: 3 disconnect: 1 [ 50s ] executions total: 4581 successful: 4578 failed: 3 disconnect: 1 [ 51s ] executions total: 4663 successful: 4660 failed: 3 disconnect: 1 [ 52s ] executions total: 4764 successful: 4761 failed: 3 disconnect: 1 [ 53s ] executions total: 4859 successful: 4856 failed: 3 disconnect: 1 [ 54s ] executions total: 4959 successful: 4956 failed: 3 disconnect: 1 [ 55s ] executions total: 5064 successful: 5061 failed: 3 disconnect: 1 [ 56s ] executions total: 5169 successful: 5166 failed: 3 disconnect: 1 [ 57s ] executions total: 5274 successful: 5271 failed: 3 disconnect: 1 [ 58s ] executions total: 5378 successful: 5375 failed: 3 disconnect: 1 [ 59s ] executions total: 5481 successful: 5478 failed: 3 disconnect: 1 [ 60s ] executions total: 5517 successful: 5514 failed: 3 disconnect: 1 Test Result: Total Executions: 5517 Successful Executions: 5514 Failed Executions: 3 Disconnection Counts: 1 Connection Information: Database Type: mysql Host: mysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local Port: 3306 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 5514 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mysql-qdhftk --namespace ns-yahzs ` pod/test-db-client-executionloop-mysql-qdhftk patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-mysql-qdhftk" force deleted LB_TYPE is set to: intranet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose mysql-qdhftk --auto-approve --force=true --type intranet --enable true --components mysql --role-selector primary --namespace ns-yahzs ` OpsRequest mysql-qdhftk-expose-f256t created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-expose-f256t -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-expose-f256t ns-yahzs Expose mysql-qdhftk mysql Running 0/1 May 28,2025 11:42 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-expose-f256t ns-yahzs Expose mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:42 UTC+0800 check ops status done ops_status:mysql-qdhftk-expose-f256t ns-yahzs Expose mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:42 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-expose-f256t --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-expose-f256t patched `kbcli cluster delete-ops --name mysql-qdhftk-expose-f256t --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-expose-f256t deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test switchover apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-switchover- namespace: ns-yahzs spec: clusterName: mysql-qdhftk type: Switchover switchover: - componentName: mysql candidateName: 'mysql-qdhftk-mysql-1' instanceName: 'mysql-qdhftk-mysql-0' `kubectl create -f test_create_mysql-qdhftk_switchover.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-switchover-hdq8q created create test_create_mysql-qdhftk_switchover.yaml Success `rm -rf test_create_mysql-qdhftk_switchover.yaml` switchover pod:mysql-qdhftk-mysql-0 checking switchover... switchover pod:mysql-qdhftk-mysql-1 switchover success LB_TYPE is set to: intranet No resources found in ns-yahzs namespace. `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-mysql-qdhftk namespace: ns-yahzs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:a1d3b58a125f442ecb390b625f8f5708-b7b564b87d08854c.elb.us-west-2.amazonaws.com,user:root,password:Ab316X41iY,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-mysql-qdhftk.yaml` pod/benchtest-mysql-qdhftk created apply benchtest-mysql-qdhftk.yaml Success `rm -rf benchtest-mysql-qdhftk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 0/1 ContainerCreating 0 24s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 36s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 42s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 48s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 54s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 60s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 66s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 71s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 77s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 83s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 89s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 95s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 101s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 107s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 113s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 119s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m11s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m17s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m23s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m29s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m35s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m41s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m47s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m53s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 2m59s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m4s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m10s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m16s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m22s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m28s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m34s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m40s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m46s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m52s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 3m58s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m4s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m10s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m16s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m22s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m28s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m34s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m40s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m45s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m51s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 4m57s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 5m3s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 5m9s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 1/1 Running 0 5m15s check pod benchtest-mysql-qdhftk status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-qdhftk 0/1 Completed 0 5m21s `kubectl logs benchtest-mysql-qdhftk --tail 30 --namespace ns-yahzs ` [ 7s ] thds: 4 tps: 53.95 qps: 1078.02 (r/w/o: 754.31/215.80/107.90) lat (ms,99%): 102.97 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 54.04 qps: 1058.87 (r/w/o: 740.61/212.17/106.09) lat (ms,99%): 150.29 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 51.00 qps: 1023.96 (r/w/o: 716.97/203.99/103.00) lat (ms,99%): 116.80 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 59.01 qps: 1209.13 (r/w/o: 847.09/243.03/119.01) lat (ms,99%): 99.33 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 7840 write: 2240 other: 1120 total: 11200 transactions: 560 (55.80 per sec.) queries: 11200 (1115.92 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0351s total number of events: 560 Latency (ms): min: 16.93 avg: 71.61 max: 179.41 99th percentile: 150.29 sum: 40101.04 Threads fairness: events (avg/stddev): 140.0000/3.67 execution time (avg/stddev): 10.0253/0.01 `kubectl delete pod benchtest-mysql-qdhftk --force --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-mysql-qdhftk" force deleted test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-qdhftk" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-mysql-qdhftk.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-mysql-qdhftk created apply test-chaos-mesh-podfailure-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-podfailure-mysql-qdhftk.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Failed May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Failed cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-qdhftk" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-mysql-qdhftk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-mysql-qdhftk --namespace ns-yahzs ` Error from server (NotFound): pods "test-db-client-connectionstress-mysql-qdhftk" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-mysql-qdhftk" not found `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-mysql-qdhftk namespace: ns-yahzs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local" - "--user" - "root" - "--password" - "Ab316X41iY" - "--port" - "3306" - "--database" - "mysql" - "--dbtype" - "mysql" - "--test" - "connectionstress" - "--connections" - "83" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-mysql-qdhftk.yaml` pod/test-db-client-connectionstress-mysql-qdhftk created apply test-db-client-connectionstress-mysql-qdhftk.yaml Success `rm -rf test-db-client-connectionstress-mysql-qdhftk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-mysql-qdhftk 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-mysql-qdhftk 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-mysql-qdhftk 1/1 Running 0 17s check pod test-db-client-connectionstress-mysql-qdhftk status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-mysql-qdhftk 0/1 Completed 0 23s check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done --host mysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local --user root --password Ab316X41iY --port 3306 --database mysql --dbtype mysql --test connectionstress --connections 83 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 03:53:47.996 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Test Result: null Connection Information: Database Type: mysql Host: mysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local Port: 3306 Database: mysql Table: User: root Org: Access Mode: mysql Test Type: connectionstress Connection Count: 83 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-mysql-qdhftk --namespace ns-yahzs ` pod/test-db-client-connectionstress-mysql-qdhftk patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-mysql-qdhftk" force deleted check failover pod name failover pod name:mysql-qdhftk-mysql-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster does not need to check monitor currently check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-qdhftk" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-mysql-qdhftk.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-mysql-qdhftk created apply test-chaos-mesh-timeoffset-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-timeoffset-mysql-qdhftk.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-qdhftk" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster configure component_tmp: mysql apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-reconfiguring- namespace: ns-yahzs spec: type: Reconfiguring clusterName: mysql-qdhftk force: true reconfigures: - componentName: mysql parameters: - key: binlog_expire_logs_seconds value: '691200' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-reconfiguring-82pq9 created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-reconfiguring-82pq9 ns-yahzs Reconfiguring mysql-qdhftk mysql,mysql Running -/- May 28,2025 11:57 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:10Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-reconfiguring-82pq9 ns-yahzs Reconfiguring mysql-qdhftk mysql,mysql Succeed -/- May 28,2025 11:57 UTC+0800 check ops status done ops_status:mysql-qdhftk-reconfiguring-82pq9 ns-yahzs Reconfiguring mysql-qdhftk mysql,mysql Succeed -/- May 28,2025 11:57 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-reconfiguring-82pq9 --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-reconfiguring-82pq9 patched `kbcli cluster delete-ops --name mysql-qdhftk-reconfiguring-82pq9 --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-reconfiguring-82pq9 deleted component_config:mysql check config variables `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'binlog_expire_logs_seconds'\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash | (grep "binlog_expire_logs_seconds" || true) | awk '***print mysql***' current value_actual: 691200 configure:[binlog_expire_logs_seconds] result actual:[691200] equal expected:[691200] `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=mysql-qdhftk,apps.kubeblocks.io/component-name=mysql,apps.kubeblocks.io/vct-name=data --namespace ns-yahzs ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in mysql-qdhftk namespace. `kbcli cluster volume-expand mysql-qdhftk --auto-approve --force=true --components mysql --volume-claim-templates data --storage 11Gi --namespace ns-yahzs ` OpsRequest mysql-qdhftk-volumeexpansion-cvmz5 created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-volumeexpansion-cvmz5 -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-volumeexpansion-cvmz5 ns-yahzs VolumeExpansion mysql-qdhftk mysql Running 0/2 May 28,2025 11:58 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done No resources found in mysql-qdhftk namespace. check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-volumeexpansion-cvmz5 ns-yahzs VolumeExpansion mysql-qdhftk mysql Succeed 2/2 May 28,2025 11:58 UTC+0800 check ops status done ops_status:mysql-qdhftk-volumeexpansion-cvmz5 ns-yahzs VolumeExpansion mysql-qdhftk mysql Succeed 2/2 May 28,2025 11:58 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-volumeexpansion-cvmz5 --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-volumeexpansion-cvmz5 patched `kbcli cluster delete-ops --name mysql-qdhftk-volumeexpansion-cvmz5 --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-volumeexpansion-cvmz5 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover networkloss check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkloss-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkloss-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkloss-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkloss-mysql-qdhftk created apply test-chaos-mesh-networkloss-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkloss-mysql-qdhftk.yaml` networkloss chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role No resources found in ns-yahzs namespace. primary: mysql-qdhftk-mysql-0 mysql-qdhftk-mysql-1;secondary: No resources found in ns-yahzs namespace. primary: mysql-qdhftk-mysql-0 mysql-qdhftk-mysql-1;secondary: check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkloss-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-mysql-qdhftk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover networkloss Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-mysql-qdhftk created apply test-chaos-mesh-networkdelay-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkdelay-mysql-qdhftk.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-qdhftk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-qdhftk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-mysql-qdhftk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-mysql-qdhftk created apply test-chaos-mesh-dnserror-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-dnserror-mysql-qdhftk.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 11:38 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-qdhftk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod mysql-qdhftk-mysql-1 --namespace ns-yahzs -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-13-226.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod mysql-qdhftk-mysql-0 --namespace ns-yahzs -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-5-43.us-west-2.compute.internal success kubectl drain ip-172-31-13-226.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-13-226.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-tp4jn, kube-system/aws-node-mrmqt, kube-system/ebs-csi-node-hx55f, kube-system/kube-proxy-gcjms evicting pod ns-yahzs/mysql-qdhftk-mysql-1 evicting pod kube-system/ebs-csi-controller-fcc5bd554-66wcv evicting pod ns-sxczu/apemysql-hmdmte-mysql-2 pod/mysql-qdhftk-mysql-1 evicted pod/apemysql-hmdmte-mysql-2 evicted pod/ebs-csi-controller-fcc5bd554-66wcv evicted node/ip-172-31-13-226.us-west-2.compute.internal drained kubectl uncordon ip-172-31-13-226.us-west-2.compute.internal node/ip-172-31-13-226.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:07 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check failover pod name failover pod name:mysql-qdhftk-mysql-0 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster mysql scale-out check cluster status before ops check cluster status done cluster_status:Running No resources found in mysql-qdhftk namespace. `kbcli cluster scale-out mysql-qdhftk --auto-approve --force=true --components mysql --replicas 1 --namespace ns-yahzs ` OpsRequest mysql-qdhftk-horizontalscaling-4zb9n created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-horizontalscaling-4zb9n -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-horizontalscaling-4zb9n ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:09 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:07 UTC+0800 mysql-qdhftk-mysql-2 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 12:09 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 mysql-qdhftk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done No resources found in mysql-qdhftk namespace. check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-horizontalscaling-4zb9n ns-yahzs HorizontalScaling mysql-qdhftk mysql Succeed 1/1 May 28,2025 12:09 UTC+0800 check ops status done ops_status:mysql-qdhftk-horizontalscaling-4zb9n ns-yahzs HorizontalScaling mysql-qdhftk mysql Succeed 1/1 May 28,2025 12:09 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-horizontalscaling-4zb9n --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-horizontalscaling-4zb9n patched `kbcli cluster delete-ops --name mysql-qdhftk-horizontalscaling-4zb9n --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-horizontalscaling-4zb9n deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster mysql scale-in check cluster status before ops check cluster status done cluster_status:Running No resources found in mysql-qdhftk namespace. `kbcli cluster scale-in mysql-qdhftk --auto-approve --force=true --components mysql --replicas 1 --namespace ns-yahzs ` OpsRequest mysql-qdhftk-horizontalscaling-8zhtj created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-horizontalscaling-8zhtj -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:07 UTC+0800 mysql-qdhftk-mysql-2 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 12:09 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 mysql-qdhftk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done No resources found in mysql-qdhftk namespace. check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Running 0/1 May 28,2025 12:11 UTC+0800 check ops status done ops_status:mysql-qdhftk-horizontalscaling-8zhtj ns-yahzs HorizontalScaling mysql-qdhftk mysql Succeed 1/1 May 28,2025 12:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-horizontalscaling-8zhtj --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-horizontalscaling-8zhtj patched `kbcli cluster delete-ops --name mysql-qdhftk-horizontalscaling-8zhtj --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-horizontalscaling-8zhtj deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-mysql-qdhftk created apply test-chaos-mesh-networkbandwidthover-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-mysql-qdhftk.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:07 UTC+0800 check pod status done check cluster role No resources found in ns-yahzs namespace. primary: mysql-qdhftk-mysql-0 mysql-qdhftk-mysql-1;secondary: No resources found in ns-yahzs namespace. primary: mysql-qdhftk-mysql-0 mysql-qdhftk-mysql-1;secondary: check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-qdhftk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-mysql-qdhftk created apply test-chaos-mesh-networkduplicate-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkduplicate-mysql-qdhftk.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:37 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 500m / 500m 512Mi / 512Mi data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:07 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-qdhftk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale mysql-qdhftk --auto-approve --force=true --components mysql --cpu 600m --memory 0.6Gi --namespace ns-yahzs ` OpsRequest mysql-qdhftk-verticalscaling-4ntmb created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-verticalscaling-4ntmb -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-verticalscaling-4ntmb ns-yahzs VerticalScaling mysql-qdhftk mysql Running 0/2 May 28,2025 12:18 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-verticalscaling-4ntmb ns-yahzs VerticalScaling mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:18 UTC+0800 check ops status done ops_status:mysql-qdhftk-verticalscaling-4ntmb ns-yahzs VerticalScaling mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:18 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-verticalscaling-4ntmb --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-verticalscaling-4ntmb patched `kbcli cluster delete-ops --name mysql-qdhftk-verticalscaling-4ntmb --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-verticalscaling-4ntmb deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mysql-qdhftk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mysql-qdhftk created apply test-chaos-mesh-oom-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-oom-mysql-qdhftk.yaml` check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mysql-qdhftk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mysql-qdhftk created apply test-chaos-mesh-oom-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-oom-mysql-qdhftk.yaml` check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found failover pod name:mysql-qdhftk-mysql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mysql-qdhftk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mysql-qdhftk created apply test-chaos-mesh-oom-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-oom-mysql-qdhftk.yaml` check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Failed May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Failed cluster_status:Failed cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-qdhftk" not found check failover pod name:mysql-qdhftk-mysql-1 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mysql-qdhftk created apply test-chaos-mesh-networkcorruptover-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-mysql-qdhftk.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found check failover pod name get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get mysql primary pod name... get pod name timeout failover pod name: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mysql-qdhftk created apply test-chaos-mesh-networkcorruptover-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-mysql-qdhftk.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-qdhftk" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mysql-qdhftk patched failover pod name:mysql-qdhftk-mysql-0 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cmpv upgrade service version:2,5.7.44|2,8.0.30|2,8.0.31|2,8.0.32|2,8.0.33|2,8.0.34|2,8.0.35|2,8.0.36|2,8.0.37|2,8.0.38|2,8.0.39 set latest cmpv service version latest service version:8.0.39 cmpv service version upgrade and downgrade 8.0.30 is less than 8.0.31 upgrade from:8.0.30 to service version:8.0.31 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.31 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-gpkmc created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-gpkmc ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:31 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-gpkmc ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:31 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-gpkmc ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-gpkmc --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-gpkmc patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-gpkmc --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-gpkmc deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success 8.0.31 is less than 8.0.32 upgrade from:8.0.31 to service version:8.0.32 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.32 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-px8rl created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-px8rl ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:34 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-px8rl ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:34 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-px8rl ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:34 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-px8rl --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-px8rl patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-px8rl --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-px8rl deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success 8.0.32 is less than 8.0.33 upgrade from:8.0.32 to service version:8.0.33 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.33 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-tqhjb created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-tqhjb ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:38 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-tqhjb ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:38 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-tqhjb ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-tqhjb --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-tqhjb patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-tqhjb --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-tqhjb deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success 8.0.33 is less than 8.0.34 upgrade from:8.0.33 to service version:8.0.34 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.34 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-bnw9c created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-bnw9c ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:41 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-bnw9c ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:41 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-bnw9c ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-bnw9c --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-bnw9c patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-bnw9c --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-bnw9c deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.34 to service version:8.0.35 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.35 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-vtbxt created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-vtbxt ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:44 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-vtbxt ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:44 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-vtbxt ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-vtbxt --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-vtbxt patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-vtbxt --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-vtbxt deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.35 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-mkszn created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-mkszn ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:47 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:18 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-mkszn ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:47 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-mkszn ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-mkszn --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-mkszn patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-mkszn --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-mkszn deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.35 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.35 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-mz2nr created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-mz2nr ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:50 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-mz2nr ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:50 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-mz2nr ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:50 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-mz2nr --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-mz2nr patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-mz2nr --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-mz2nr deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.35 to service version:8.0.36 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.36 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-lww2b created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-lww2b ns-yahzs Upgrade mysql-qdhftk Running -/- May 28,2025 12:52 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-lww2b ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:52 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-lww2b ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-lww2b --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-lww2b patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-lww2b --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-lww2b deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.36 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-2mzc7 created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-2mzc7 ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:55 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-2mzc7 ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:55 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-2mzc7 ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:55 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-2mzc7 --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-2mzc7 patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-2mzc7 --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-2mzc7 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.36 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.36 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-89t9t created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-89t9t ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 12:58 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-89t9t ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:58 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-89t9t ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 12:58 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-89t9t --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-89t9t patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-89t9t --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-89t9t deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.36 to service version:8.0.37 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.37 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-54cvw created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-54cvw ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 13:00 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-54cvw ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:00 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-54cvw ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:00 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-54cvw --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-54cvw patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-54cvw --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-54cvw deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.37 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-gv67s created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-gv67s ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 13:03 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-gv67s ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:03 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-gv67s ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:03 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-gv67s --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-gv67s patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-gv67s --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-gv67s deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.37 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.37 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-w7k5l created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-w7k5l ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 13:06 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-w7k5l ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:06 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-w7k5l ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:06 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-w7k5l --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-w7k5l patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-w7k5l --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-w7k5l deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.37 to service version:8.0.38 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.38 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-vrm5t created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-vrm5t ns-yahzs Upgrade mysql-qdhftk Creating -/- May 28,2025 13:08 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-vrm5t ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:08 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-vrm5t ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-vrm5t --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-vrm5t patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-vrm5t --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-vrm5t deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success upgrade from:8.0.38 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-tx64t created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-tx64t ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 13:11 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-tx64t ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:11 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-tx64t ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-tx64t --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-tx64t patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-tx64t --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-tx64t deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.38 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-upgrade-cmpv- namespace: ns-yahzs spec: clusterName: mysql-qdhftk upgrade: components: - componentName: mysql serviceVersion: 8.0.38 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-79ssv created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-79ssv ns-yahzs Upgrade mysql-qdhftk mysql Running 0/2 May 28,2025 13:13 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:50 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 12:20 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-upgrade-cmpv-79ssv ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:13 UTC+0800 check ops status done ops_status:mysql-qdhftk-upgrade-cmpv-79ssv ns-yahzs Upgrade mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:13 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-upgrade-cmpv-79ssv --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-upgrade-cmpv-79ssv patched `kbcli cluster delete-ops --name mysql-qdhftk-upgrade-cmpv-79ssv --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-upgrade-cmpv-79ssv deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart mysql-qdhftk --auto-approve --force=true --namespace ns-yahzs ` OpsRequest mysql-qdhftk-restart-w2qqz created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-restart-w2qqz -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-restart-w2qqz ns-yahzs Restart mysql-qdhftk mysql Running 0/2 May 28,2025 13:15 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 13:17 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-restart-w2qqz ns-yahzs Restart mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:15 UTC+0800 check ops status done ops_status:mysql-qdhftk-restart-w2qqz ns-yahzs Restart mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:15 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-restart-w2qqz --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-restart-w2qqz patched `kbcli cluster delete-ops --name mysql-qdhftk-restart-w2qqz --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-restart-w2qqz deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-qdhftk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-mysql-qdhftk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-mysql-qdhftk created apply test-chaos-mesh-dnsrandom-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-dnsrandom-mysql-qdhftk.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 13:17 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-qdhftk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-qdhftk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 action: partition mode: all target: mode: all selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-mysql-qdhftk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-mysql-qdhftk created apply test-chaos-mesh-networkpartition-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-networkpartition-mysql-qdhftk.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 13:17 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-qdhftk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-qdhftk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-qdhftk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-mysql-qdhftk namespace: ns-yahzs spec: selector: namespaces: - ns-yahzs labelSelectors: apps.kubeblocks.io/pod-name: mysql-qdhftk-mysql-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-mysql-qdhftk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-mysql-qdhftk created apply test-chaos-mesh-fullcpu-mysql-qdhftk.yaml Success `rm -rf test-chaos-mesh-fullcpu-mysql-qdhftk.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 13:17 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-1;secondary: mysql-qdhftk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mysql-qdhftk --namespace ns-yahzs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-qdhftk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-qdhftk" not found check failover pod name failover pod name:mysql-qdhftk-mysql-1 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop mysql-qdhftk --auto-approve --force=true --namespace ns-yahzs ` OpsRequest mysql-qdhftk-stop-rrpdn created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-stop-rrpdn -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-stop-rrpdn ns-yahzs Stop mysql-qdhftk mysql Running 0/2 May 28,2025 13:26 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Stopping May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-stop-rrpdn ns-yahzs Stop mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:26 UTC+0800 check ops status done ops_status:mysql-qdhftk-stop-rrpdn ns-yahzs Stop mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:26 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-stop-rrpdn --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-stop-rrpdn patched `kbcli cluster delete-ops --name mysql-qdhftk-stop-rrpdn --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-stop-rrpdn deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start mysql-qdhftk --force=true --namespace ns-yahzs ` OpsRequest mysql-qdhftk-start-zczrc created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-start-zczrc -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-start-zczrc ns-yahzs Start mysql-qdhftk mysql Running 0/2 May 28,2025 13:27 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql Delete Updating May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:27 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 13:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-start-zczrc ns-yahzs Start mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:27 UTC+0800 check ops status done ops_status:mysql-qdhftk-start-zczrc ns-yahzs Start mysql-qdhftk mysql Succeed 2/2 May 28,2025 13:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-start-zczrc --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-start-zczrc patched `kbcli cluster delete-ops --name mysql-qdhftk-start-zczrc --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-start-zczrc deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update mysql-qdhftk --termination-policy=WipeOut --namespace ns-yahzs ` cluster.apps.kubeblocks.io/mysql-qdhftk updated check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql WipeOut Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:27 UTC+0800 mysql-qdhftk-mysql-1 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 13:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done cluster xtrabackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mysql-qdhftk --method xtrabackup --namespace ns-yahzs ` Backup backup-ns-yahzs-mysql-qdhftk-20250528133059 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-yahzs-mysql-qdhftk-20250528133059 -n ns-yahzs check backup status `kbcli cluster list-backups mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-yahzs-mysql-qdhftk-20250528133059 ns-yahzs mysql-qdhftk xtrabackup Running Delete May 28,2025 13:30 UTC+0800 backup_status:mysql-qdhftk-xtrabackup-Running backup_status:mysql-qdhftk-xtrabackup-Running backup_status:mysql-qdhftk-xtrabackup-Running check backup status done backup_status:backup-ns-yahzs-mysql-qdhftk-20250528133059 ns-yahzs mysql-qdhftk xtrabackup Completed 9458310 15s Delete May 28,2025 13:30 UTC+0800 May 28,2025 13:31 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "abk-qdhftk" not found `kbcli cluster describe-backup --names backup-ns-yahzs-mysql-qdhftk-20250528133059 --namespace ns-yahzs ` Name: backup-ns-yahzs-mysql-qdhftk-20250528133059 Cluster: mysql-qdhftk Namespace: ns-yahzs Spec: Method: xtrabackup Policy Name: mysql-qdhftk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-yahzs-mysql-qdhftk-20250528133059-dc960b1 TargetPodName: mysql-qdhftk-mysql-1 Phase: Completed Start Time: May 28,2025 13:31 UTC+0800 Completion Time: May 28,2025 13:31 UTC+0800 Status: Phase: Completed Total Size: 9458310 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 15s Start Time: May 28,2025 13:31 UTC+0800 Completion Time: May 28,2025 13:31 UTC+0800 Path: /ns-yahzs/mysql-qdhftk-76560ff6-eda2-49d7-b74c-48f5b49259cb/mysql/backup-ns-yahzs-mysql-qdhftk-20250528133059 Warning Events: `kbcli cluster restore abk-qdhftk --backup backup-ns-yahzs-mysql-qdhftk-20250528133059 --namespace ns-yahzs ` Cluster abk-qdhftk created check cluster status `kbcli cluster list abk-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS abk-qdhftk ns-yahzs mysql WipeOut Creating May 28,2025 13:31 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances abk-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME abk-qdhftk-mysql-0 ns-yahzs abk-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 13:31 UTC+0800 abk-qdhftk-mysql-1 ns-yahzs abk-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:32 UTC+0800 check pod status done check cluster role check cluster role done primary: abk-qdhftk-mysql-0;secondary: abk-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it abk-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kbcli cluster describe-backup --names backup-ns-yahzs-mysql-qdhftk-20250528133059 --namespace ns-yahzs ` Name: backup-ns-yahzs-mysql-qdhftk-20250528133059 Cluster: mysql-qdhftk Namespace: ns-yahzs Spec: Method: xtrabackup Policy Name: mysql-qdhftk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-yahzs-mysql-qdhftk-20250528133059-dc960b1 TargetPodName: mysql-qdhftk-mysql-1 Phase: Completed Start Time: May 28,2025 13:31 UTC+0800 Completion Time: May 28,2025 13:31 UTC+0800 Status: Phase: Completed Total Size: 9458310 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 15s Start Time: May 28,2025 13:31 UTC+0800 Completion Time: May 28,2025 13:31 UTC+0800 Path: /ns-yahzs/mysql-qdhftk-76560ff6-eda2-49d7-b74c-48f5b49259cb/mysql/backup-ns-yahzs-mysql-qdhftk-20250528133059 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: abk-qdhftk-mysql-0.abk-qdhftk-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 1 Master_Log_File: abk-qdhftk-mysql-0-bin.000001 Read_Master_Log_Pos: 2944 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 3186 Relay_Master_Log_File: abk-qdhftk-mysql-0-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 2944 Relay_Log_Space: 3390 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 0e30ebde-3b85-11f0-8b79-1abe297599bd Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 0e30ebde-3b85-11f0-8b79-1abe297599bd:1-8 Executed_Gtid_Set: 0e30ebde-3b85-11f0-8b79-1abe297599bd:1-8, 38cf5c64-3b75-11f0-bf1d-6e9affc73c51:1-5551, 5835508f-3b75-11f0-b75c-b24528bf18e2:1-581 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 1 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pAb316X41iY -P3306 " | kubectl exec -it abk-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster abk-qdhftk `kbcli cluster delete abk-qdhftk --auto-approve --namespace ns-yahzs ` Cluster abk-qdhftk deleted pod_info:abk-qdhftk-mysql-0 3/3 Terminating 0 77s abk-qdhftk-mysql-1 3/3 Terminating 0 54s pod_info:abk-qdhftk-mysql-0 3/3 Terminating 0 98s abk-qdhftk-mysql-1 3/3 Terminating 0 75s No resources found in ns-yahzs namespace. delete cluster pod done No resources found in ns-yahzs namespace. check cluster resource non-exist OK: pvc No resources found in ns-yahzs namespace. delete cluster done No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-qdhftk-rebuildinstance- namespace: ns-yahzs spec: type: RebuildInstance clusterName: mysql-qdhftk force: true rebuildFrom: - componentName: mysql instances: - name: mysql-qdhftk-mysql-1 backupName: backup-ns-yahzs-mysql-qdhftk-20250528133059 inPlace: false restoreEnv: - name: BACKUP_FOR_STANDBY value: "true" check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-qdhftk.yaml` opsrequest.operations.kubeblocks.io/mysql-qdhftk-rebuildinstance-pjx7g created create test_ops_cluster_mysql-qdhftk.yaml Success `rm -rf test_ops_cluster_mysql-qdhftk.yaml` check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Running 0/1 May 28,2025 13:33 UTC+0800 check ops status done ops_status:mysql-qdhftk-rebuildinstance-pjx7g ns-yahzs RebuildInstance mysql-qdhftk mysql Succeed 1/1 May 28,2025 13:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-rebuildinstance-pjx7g --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-rebuildinstance-pjx7g patched `kbcli cluster delete-ops --name mysql-qdhftk-rebuildinstance-pjx7g --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-rebuildinstance-pjx7g deleted check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql WipeOut Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:27 UTC+0800 mysql-qdhftk-mysql-2 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:33 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-2 --namespace ns-yahzs -- bash ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-2 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-yahzs-mysql-qdhftk-20250528133059 --namespace ns-yahzs ` backup.dataprotection.kubeblocks.io/backup-ns-yahzs-mysql-qdhftk-20250528133059 patched `kbcli cluster delete-backup mysql-qdhftk --name backup-ns-yahzs-mysql-qdhftk-20250528133059 --force --auto-approve --namespace ns-yahzs ` Backup backup-ns-yahzs-mysql-qdhftk-20250528133059 deleted cluster xtrabackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mysql-qdhftk --method xtrabackup --namespace ns-yahzs ` Backup backup-ns-yahzs-mysql-qdhftk-20250528133613 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-yahzs-mysql-qdhftk-20250528133613 -n ns-yahzs check backup status `kbcli cluster list-backups mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-yahzs-mysql-qdhftk-20250528133613 ns-yahzs mysql-qdhftk xtrabackup Running Delete May 28,2025 13:36 UTC+0800 backup_status:mysql-qdhftk-xtrabackup-Running backup_status:mysql-qdhftk-xtrabackup-Running backup_status:mysql-qdhftk-xtrabackup-Running check backup status done backup_status:backup-ns-yahzs-mysql-qdhftk-20250528133613 ns-yahzs mysql-qdhftk xtrabackup Completed 5811258 20s Delete May 28,2025 13:36 UTC+0800 May 28,2025 13:36 UTC+0800 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql `CREATE DATABASE IF NOT EXISTS kbclidb;CREATE TABLE IF NOT EXISTS kbclidb.msg (id INT UNSIGNED NOT NULL AUTO_INCREMENT,msg TEXT,time TIMESTAMP NULL,PRIMARY KEY (id)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;insert into kbclidb.msg (msg, time) values ('kbcli-test-data-qdhftk0', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql `insert into kbclidb.msg (msg, time) values ('kbcli-test-data-qdhftk1', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. ERROR 1290 (HY000) at line 1: The MySQL server is running with the --super-read-only option so it cannot execute this statement command terminated with exit code 1 Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql `insert into kbclidb.msg (msg, time) values ('kbcli-test-data-qdhftk2', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql `insert into kbclidb.msg (msg, time) values ('kbcli-test-data-qdhftk3', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. ERROR 1290 (HY000) at line 1: The MySQL server is running with the --super-read-only option so it cannot execute this statement command terminated with exit code 1 Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. cluster xtrabackup-inc backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mysql-qdhftk --method xtrabackup-inc --namespace ns-yahzs ` Backup backup-ns-yahzs-mysql-qdhftk-20250528133743 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-yahzs-mysql-qdhftk-20250528133743 -n ns-yahzs check backup status `kbcli cluster list-backups mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-yahzs-mysql-qdhftk-20250528133613 ns-yahzs mysql-qdhftk xtrabackup Completed 5811258 20s Delete May 28,2025 13:36 UTC+0800 May 28,2025 13:36 UTC+0800 backup-ns-yahzs-mysql-qdhftk-20250528133743 ns-yahzs mysql-qdhftk xtrabackup-inc Running Delete May 28,2025 13:37 UTC+0800 backup_status:mysql-qdhftk-xtrabackup-inc-Running backup_status:mysql-qdhftk-xtrabackup-inc-Running check backup status done backup_status:backup-ns-yahzs-mysql-qdhftk-20250528133743 ns-yahzs mysql-qdhftk xtrabackup-inc Completed 163025 10s Delete May 28,2025 13:37 UTC+0800 May 28,2025 13:37 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "abk-qdhftk" not found `kbcli cluster describe-backup --names backup-ns-yahzs-mysql-qdhftk-20250528133743 --namespace ns-yahzs ` Name: backup-ns-yahzs-mysql-qdhftk-20250528133743 Cluster: mysql-qdhftk Namespace: ns-yahzs Spec: Method: xtrabackup-inc Policy Name: mysql-qdhftk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-yahzs-mysql-qdhftk-20250528133743-8097f1f TargetPodName: mysql-qdhftk-mysql-2 Phase: Completed Start Time: May 28,2025 13:37 UTC+0800 Completion Time: May 28,2025 13:37 UTC+0800 Status: Phase: Completed Total Size: 163025 ActionSet Name: mysql-xtrabackup-inc-br Repository: backuprepo-kbcli-test Duration: 10s Start Time: May 28,2025 13:37 UTC+0800 Completion Time: May 28,2025 13:37 UTC+0800 Path: /ns-yahzs/mysql-qdhftk-76560ff6-eda2-49d7-b74c-48f5b49259cb/mysql/backup-ns-yahzs-mysql-qdhftk-20250528133743 Warning Events: `kbcli cluster restore abk-qdhftk --backup backup-ns-yahzs-mysql-qdhftk-20250528133743 --namespace ns-yahzs ` Cluster abk-qdhftk created check cluster status `kbcli cluster list abk-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS abk-qdhftk ns-yahzs mysql WipeOut Creating May 28,2025 13:38 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances abk-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME abk-qdhftk-mysql-0 ns-yahzs abk-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 13:38 UTC+0800 abk-qdhftk-mysql-1 ns-yahzs abk-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 13:39 UTC+0800 check pod status done check cluster role check cluster role done primary: abk-qdhftk-mysql-0;secondary: abk-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it abk-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kbcli cluster describe-backup --names backup-ns-yahzs-mysql-qdhftk-20250528133743 --namespace ns-yahzs ` Name: backup-ns-yahzs-mysql-qdhftk-20250528133743 Cluster: mysql-qdhftk Namespace: ns-yahzs Spec: Method: xtrabackup-inc Policy Name: mysql-qdhftk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-yahzs-mysql-qdhftk-20250528133743-8097f1f TargetPodName: mysql-qdhftk-mysql-2 Phase: Completed Start Time: May 28,2025 13:37 UTC+0800 Completion Time: May 28,2025 13:37 UTC+0800 Status: Phase: Completed Total Size: 163025 ActionSet Name: mysql-xtrabackup-inc-br Repository: backuprepo-kbcli-test Duration: 10s Start Time: May 28,2025 13:37 UTC+0800 Completion Time: May 28,2025 13:37 UTC+0800 Path: /ns-yahzs/mysql-qdhftk-76560ff6-eda2-49d7-b74c-48f5b49259cb/mysql/backup-ns-yahzs-mysql-qdhftk-20250528133743 Warning Events: `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql `select * from kbclidb.msg;` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. id msg time 1 kbcli-test-data-qdhftk0 2025-05-28 05:36:43 2 kbcli-test-data-qdhftk2 2025-05-28 05:37:07 Point-In-Time Recovery Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql cluster connect `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: abk-qdhftk-mysql-0.abk-qdhftk-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 1 Master_Log_File: abk-qdhftk-mysql-0-bin.000001 Read_Master_Log_Pos: 2944 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 3186 Relay_Master_Log_File: abk-qdhftk-mysql-0-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 2944 Relay_Log_Space: 3390 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 00e3a010-3b86-11f0-89c4-de2e1f3ac4c7 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 00e3a010-3b86-11f0-89c4-de2e1f3ac4c7:1-8 Executed_Gtid_Set: 00e3a010-3b86-11f0-89c4-de2e1f3ac4c7:1-8, 38cf5c64-3b75-11f0-bf1d-6e9affc73c51:1-5555, 5835508f-3b75-11f0-b75c-b24528bf18e2:1-581 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 1 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pAb316X41iY -P3306 " | kubectl exec -it abk-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster abk-qdhftk `kbcli cluster delete abk-qdhftk --auto-approve --namespace ns-yahzs ` Cluster abk-qdhftk deleted pod_info:abk-qdhftk-mysql-0 3/3 Terminating 0 2m41s abk-qdhftk-mysql-1 3/3 Terminating 0 72s pod_info:abk-qdhftk-mysql-0 3/3 Terminating 0 3m2s abk-qdhftk-mysql-1 3/3 Terminating 0 93s No resources found in ns-yahzs namespace. delete cluster pod done No resources found in ns-yahzs namespace. check cluster resource non-exist OK: pvc No resources found in ns-yahzs namespace. delete cluster done No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-yahzs-mysql-qdhftk-20250528133613 --namespace ns-yahzs ` backup.dataprotection.kubeblocks.io/backup-ns-yahzs-mysql-qdhftk-20250528133613 patched `kbcli cluster delete-backup mysql-qdhftk --name backup-ns-yahzs-mysql-qdhftk-20250528133613 --force --auto-approve --namespace ns-yahzs ` Backup backup-ns-yahzs-mysql-qdhftk-20250528133613 deleted `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-yahzs-mysql-qdhftk-20250528133743 --namespace ns-yahzs ` backup.dataprotection.kubeblocks.io/backup-ns-yahzs-mysql-qdhftk-20250528133743 patched `kbcli cluster delete-backup mysql-qdhftk --name backup-ns-yahzs-mysql-qdhftk-20250528133743 --force --auto-approve --namespace ns-yahzs ` Backup backup-ns-yahzs-mysql-qdhftk-20250528133743 deleted `kubectl get backupschedule -l app.kubernetes.io/instance=mysql-qdhftk ` `kubectl get backupschedule mysql-qdhftk-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched check backup status `kbcli cluster list-backups mysql-qdhftk --namespace ns-yahzs ` No backups found in ns-yahzs namespace. No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- No backups found in ns-yahzs namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=mysql-qdhftk ` `kubectl get backupschedule mysql-qdhftk-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backup_status:mysql-qdhftk-xtrabackup-Running check backup status done backup_status:mysql-qdhftk-xtrabackup-20250528054300 ns-yahzs mysql-qdhftk xtrabackup Completed 5822419 10s Delete May 28,2025 13:43 UTC+0800 May 28,2025 13:43 UTC+0800 Jun 05,2025 13:43 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=mysql-qdhftk ` `kubectl get backupschedule mysql-qdhftk-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-qdhftk-mysql-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "abk-qdhftk" not found `kbcli cluster describe-backup --names mysql-qdhftk-xtrabackup-20250528054300 --namespace ns-yahzs ` Name: mysql-qdhftk-xtrabackup-20250528054300 Cluster: mysql-qdhftk Namespace: ns-yahzs Spec: Method: xtrabackup Policy Name: mysql-qdhftk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-mysql-qdhftk-xtrabackup-20250528054300-249dc6eb TargetPodName: mysql-qdhftk-mysql-2 Phase: Completed Start Time: May 28,2025 13:43 UTC+0800 Completion Time: May 28,2025 13:43 UTC+0800 Status: Phase: Completed Total Size: 5822419 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 10s Expiration Time: Jun 05,2025 13:43 UTC+0800 Start Time: May 28,2025 13:43 UTC+0800 Completion Time: May 28,2025 13:43 UTC+0800 Path: /ns-yahzs/mysql-qdhftk-76560ff6-eda2-49d7-b74c-48f5b49259cb/mysql/mysql-qdhftk-xtrabackup-20250528054300 Warning Events: `kbcli cluster restore abk-qdhftk --backup mysql-qdhftk-xtrabackup-20250528054300 --namespace ns-yahzs ` Cluster abk-qdhftk created check cluster status `kbcli cluster list abk-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS abk-qdhftk ns-yahzs mysql WipeOut Creating May 28,2025 13:43 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances abk-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME abk-qdhftk-mysql-0 ns-yahzs abk-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 13:43 UTC+0800 abk-qdhftk-mysql-1 ns-yahzs abk-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 13:45 UTC+0800 check pod status done check cluster role check cluster role done primary: abk-qdhftk-mysql-0;secondary: abk-qdhftk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it abk-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` check cluster connect done `kbcli cluster describe-backup --names mysql-qdhftk-xtrabackup-20250528054300 --namespace ns-yahzs ` Name: mysql-qdhftk-xtrabackup-20250528054300 Cluster: mysql-qdhftk Namespace: ns-yahzs Spec: Method: xtrabackup Policy Name: mysql-qdhftk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-mysql-qdhftk-xtrabackup-20250528054300-249dc6eb TargetPodName: mysql-qdhftk-mysql-2 Phase: Completed Start Time: May 28,2025 13:43 UTC+0800 Completion Time: May 28,2025 13:43 UTC+0800 Status: Phase: Completed Total Size: 5822419 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 10s Expiration Time: Jun 05,2025 13:43 UTC+0800 Start Time: May 28,2025 13:43 UTC+0800 Completion Time: May 28,2025 13:43 UTC+0800 Path: /ns-yahzs/mysql-qdhftk-76560ff6-eda2-49d7-b74c-48f5b49259cb/mysql/mysql-qdhftk-xtrabackup-20250528054300 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=abk-qdhftk` set secret: abk-qdhftk-mysql-account-root `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: abk-qdhftk-mysql-0.abk-qdhftk-mysql-headless Master_User: root Master_Port: 3306 Connect_Retry: 1 Master_Log_File: abk-qdhftk-mysql-0-bin.000001 Read_Master_Log_Pos: 2945 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 3187 Relay_Master_Log_File: abk-qdhftk-mysql-0-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 2945 Relay_Log_Space: 3391 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: No Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: c33322fa-3b86-11f0-9494-bea84115ee32 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: c33322fa-3b86-11f0-9494-bea84115ee32:1-8 Executed_Gtid_Set: 38cf5c64-3b75-11f0-bf1d-6e9affc73c51:1-5555, 5835508f-3b75-11f0-b75c-b24528bf18e2:1-581, c33322fa-3b86-11f0-9494-bea84115ee32:1-8 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 1 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pAb316X41iY -P3306 " | kubectl exec -it abk-qdhftk-mysql-1 --namespace ns-yahzs -- bash ` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster abk-qdhftk `kbcli cluster delete abk-qdhftk --auto-approve --namespace ns-yahzs ` Cluster abk-qdhftk deleted pod_info:abk-qdhftk-mysql-0 3/3 Terminating 0 2m11s abk-qdhftk-mysql-1 3/3 Terminating 0 48s pod_info:abk-qdhftk-mysql-0 3/3 Terminating 0 2m32s abk-qdhftk-mysql-1 3/3 Terminating 0 69s No resources found in ns-yahzs namespace. delete cluster pod done No resources found in ns-yahzs namespace. check cluster resource non-exist OK: pvc No resources found in ns-yahzs namespace. delete cluster done No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups mysql-qdhftk-xtrabackup-20250528054300 --namespace ns-yahzs ` backup.dataprotection.kubeblocks.io/mysql-qdhftk-xtrabackup-20250528054300 patched `kbcli cluster delete-backup mysql-qdhftk --name mysql-qdhftk-xtrabackup-20250528054300 --force --auto-approve --namespace ns-yahzs ` Backup mysql-qdhftk-xtrabackup-20250528054300 deleted cluster list-logs `kbcli cluster list-logs mysql-qdhftk --namespace ns-yahzs ` No log files found. Error from server (NotFound): pods "mysql-qdhftk-mysql-0" not found cluster logs `kbcli cluster logs mysql-qdhftk --tail 30 --namespace ns-yahzs ` Defaulted container "mysql" out of: mysql, kbagent, config-manager, init-data (init), init-jq (init), init-kbagent (init), kbagent-worker (init) 2025-05-28 05:28:14+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.38-1.el9 started. 2025-05-28 05:28:14+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql' 2025-05-28 05:28:14+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.38-1.el9 started. '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock' cluster logs slow `kbcli cluster logs mysql-qdhftk --tail 30 --file-type=slow --namespace ns-yahzs ` error: can't get file path pattern by type slow cluster logs error `kbcli cluster logs mysql-qdhftk --tail 30 --file-type=error --namespace ns-yahzs ` error: can't get file path pattern by type error LB_TYPE is set to: intranet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose mysql-qdhftk --auto-approve --force=true --type intranet --enable false --components mysql --role-selector primary --namespace ns-yahzs ` OpsRequest mysql-qdhftk-expose-r727h created successfully, you can view the progress: kbcli cluster describe-ops mysql-qdhftk-expose-r727h -n ns-yahzs check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-expose-r727h ns-yahzs Expose mysql-qdhftk mysql Running 0/1 May 28,2025 13:47 UTC+0800 check cluster status `kbcli cluster list mysql-qdhftk --show-labels --namespace ns-yahzs ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-qdhftk ns-yahzs mysql WipeOut Running May 28,2025 11:37 UTC+0800 app.kubernetes.io/instance=mysql-qdhftk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-qdhftk --namespace ns-yahzs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-qdhftk-mysql-0 ns-yahzs mysql-qdhftk mysql Running primary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:27 UTC+0800 mysql-qdhftk-mysql-2 ns-yahzs mysql-qdhftk mysql Running secondary us-west-2a 600m / 600m 644245094400m / 644245094400m data:11Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 13:33 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-qdhftk-mysql-0;secondary: mysql-qdhftk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-2 --namespace ns-yahzs -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-qdhftk --status all --namespace ns-yahzs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-qdhftk-switchover-hdq8q ns-yahzs Switchover mysql-qdhftk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 mysql-qdhftk-expose-r727h ns-yahzs Expose mysql-qdhftk mysql Succeed 1/1 May 28,2025 13:47 UTC+0800 check ops status done ops_status:mysql-qdhftk-expose-r727h ns-yahzs Expose mysql-qdhftk mysql Succeed 1/1 May 28,2025 13:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-qdhftk-expose-r727h --namespace ns-yahzs ` opsrequest.operations.kubeblocks.io/mysql-qdhftk-expose-r727h patched `kbcli cluster delete-ops --name mysql-qdhftk-expose-r727h --force --auto-approve --namespace ns-yahzs ` OpsRequest mysql-qdhftk-expose-r727h deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-qdhftk-mysql-server.ns-yahzs.svc.cluster.local -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-0 --namespace ns-yahzs -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-qdhftk` set secret: mysql-qdhftk-mysql-account-root `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-qdhftk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:Ab316X41iY;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'Ab316X41iY' " | kubectl exec -it mysql-qdhftk-mysql-2 --namespace ns-yahzs -- bash ` check readonly db_client batch data Success delete cluster mysql-qdhftk `kbcli cluster delete mysql-qdhftk --auto-approve --namespace ns-yahzs ` Cluster mysql-qdhftk deleted pod_info:mysql-qdhftk-mysql-0 3/3 Terminating 0 20m mysql-qdhftk-mysql-2 3/3 Terminating 0 14m pod_info:mysql-qdhftk-mysql-0 3/3 Terminating 0 20m mysql-qdhftk-mysql-2 3/3 Terminating 0 14m No resources found in ns-yahzs namespace. delete cluster pod done No resources found in ns-yahzs namespace. check cluster resource non-exist OK: pvc No resources found in ns-yahzs namespace. delete cluster done No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. Mysql Test Suite All Done! delete cluster orcm-qdhftk `kbcli cluster delete orcm-qdhftk --auto-approve --namespace ns-yahzs ` Cluster orcm-qdhftk deleted pod_info:orcm-qdhftk-orchestrator-0 2/2 Terminating 0 133m orcm-qdhftk-orchestrator-1 2/2 Terminating 0 132m orcm-qdhftk-orchestrator-2 2/2 Terminating 0 132m pod_info:orcm-qdhftk-orchestrator-0 2/2 Terminating 0 133m orcm-qdhftk-orchestrator-1 2/2 Terminating 0 133m orcm-qdhftk-orchestrator-2 2/2 Terminating 0 132m No resources found in ns-yahzs namespace. delete cluster pod done No resources found in ns-yahzs namespace. check cluster resource non-exist OK: pvc No resources found in ns-yahzs namespace. delete cluster done No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. No resources found in ns-yahzs namespace. [PASSED]|[Create]|[ComponentDefinition=orchestrator-raft-1.0.0-alpha.0;ServiceVersion=3.2.6;]|[Description=Create a cluster with the specified component definition orchestrator-raft-1.0.0-alpha.0 and service version 3.2.6] --------------------------------------Mysql Orchestrator (Topology = orc Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=mysql-8.0-1.0.0-alpha.0;ComponentVersion=mysql;ServiceVersion=8.0.30;]|[Description=Create a cluster with the specified component definition mysql-8.0-1.0.0-alpha.0 and component version mysql and service version 8.0.30] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Expose]|[Enable=true;TYPE=intranet;ComponentName=mysql]|[Description=Expose Enable the intranet service with mysql component] [PASSED]|[SwitchOver]|[ComponentName=mysql]|[Description=SwitchOver the cluster specify component mysql] [PASSED]|[Bench]|[HostType=LB;ComponentName=mysql]|[Description=Bench the cluster LB service with mysql component] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=mysql]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=mysql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Reconfiguring]|[ComponentName=mysql;binlog_expire_logs_seconds=691200]|[Description=Reconfiguring the cluster specify component mysql set binlog_expire_logs_seconds=691200] [PASSED]|[VolumeExpansion]|[ComponentName=mysql]|[Description=VolumeExpansion the cluster specify component mysql] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=mysql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=mysql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[HorizontalScaling Out]|[ComponentName=mysql]|[Description=HorizontalScaling Out the cluster specify component mysql] [PASSED]|[HorizontalScaling In]|[ComponentName=mysql]|[Description=HorizontalScaling In the cluster specify component mysql] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=mysql]|[Description=] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mysql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[VerticalScaling]|[ComponentName=mysql]|[Description=VerticalScaling the cluster specify component mysql] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=mysql]|[Description=] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.30;ComponentVersionTo=8.0.31]|[Description=Upgrade the cluster specify component mysql service version from 8.0.30 to 8.0.31] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.31;ComponentVersionTo=8.0.32]|[Description=Upgrade the cluster specify component mysql service version from 8.0.31 to 8.0.32] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.32;ComponentVersionTo=8.0.33]|[Description=Upgrade the cluster specify component mysql service version from 8.0.32 to 8.0.33] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.33;ComponentVersionTo=8.0.34]|[Description=Upgrade the cluster specify component mysql service version from 8.0.33 to 8.0.34] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.34;ComponentVersionTo=8.0.35]|[Description=Upgrade the cluster specify component mysql service version from 8.0.34 to 8.0.35] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.35;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.35 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.35]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.35] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.35;ComponentVersionTo=8.0.36]|[Description=Upgrade the cluster specify component mysql service version from 8.0.35 to 8.0.36] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.36;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.36 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.36]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.36] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.36;ComponentVersionTo=8.0.37]|[Description=Upgrade the cluster specify component mysql service version from 8.0.36 to 8.0.37] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.37;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.37 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.37]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.37] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.37;ComponentVersionTo=8.0.38]|[Description=Upgrade the cluster specify component mysql service version from 8.0.37 to 8.0.38] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.38;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.38 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.38]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.38] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=mysql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Backup] [PASSED]|[Restore]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=xtrabackup]|[Description=Delete the xtrabackup restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=mysql]|[Description=Rebuild the cluster instance specify component mysql] [PASSED]|[Backup]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Backup] [PASSED]|[Backup]|[BackupMethod=xtrabackup-inc]|[Description=The cluster xtrabackup-inc Backup] [PASSED]|[Restore Increment]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Restore Increment] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=xtrabackup]|[Description=Delete the xtrabackup restore cluster] [PASSED]|[Backup]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=xtrabackup]|[Description=Delete the Schedule xtrabackup restore cluster] [PASSED]|[Expose]|[Disable=true;TYPE=intranet;ComponentName=mysql]|[Description=Expose Disable the intranet service with mysql component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]