source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-cbpst ` `kubectl create namespace ns-cbpst` namespace/ns-cbpst created create namespace ns-cbpst done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.0` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 48 33.6M 48 16.4M 0 0 86.9M 0 --:--:-- --:--:-- --:--:-- 86.9M 100 33.6M 100 33.6M 0 0 124M 0 --:--:-- --:--:-- --:--:-- 215M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.0 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Kubernetes Env: v1.32.5-eks-5d4a308 check snapshot controller check snapshot controller done eks default-vsc found POD_RESOURCES: No resources found found default storage class: gp3 KubeBlocks version is:1.0.0 skip upgrade KubeBlocks current KubeBlocks version: 1.0.0 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:mysql set component version set component version:mysql set service versions:8.4.2,8.4.1,8.4.0,8.0.39,8.0.38,8.0.37,8.0.36,8.0.35,8.0.34,8.0.33,8.0.32,8.0.31,8.0.30,5.7.44 set service versions sorted:5.7.44,8.0.30,8.0.31,8.0.32,8.0.33,8.0.34,8.0.35,8.0.36,8.0.37,8.0.38,8.0.39,8.4.0,8.4.1,8.4.2 set mysql component definition set mysql component definition set replicas first:2,5.7.44|2,8.0.30|2,8.0.31|2,8.0.32|2,8.0.33|2,8.0.34|2,8.0.35|2,8.0.36|2,8.0.37|2,8.0.38|2,8.0.39|2,8.4.0|2,8.4.1|2,8.4.2 set replicas third:2,8.0.32 set replicas fourth:2,8.0.30 set minimum cmpv service version set minimum cmpv service version replicas:2,8.0.30 REPORT_COUNT:1 CLUSTER_TOPOLOGY:semisync topology semisync found in cluster definition mysql set mysql component definition set mysql component definition LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 10 No resources found in ns-cbpst namespace. termination_policy:WipeOut create 2 replica WipeOut mysql cluster check component definition set component definition by component version check cmpd by labels set component definition1: mysql-8.0-1.0.0-alpha.0 by component version:mysql apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: mysql-fphydk namespace: ns-cbpst spec: clusterDef: mysql topology: semisync terminationPolicy: WipeOut componentSpecs: - name: mysql serviceVersion: 8.0.30 disableExporter: true replicas: 2 resources: limits: cpu: 100m memory: 0.5Gi requests: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi `kubectl apply -f test_create_mysql-fphydk.yaml` cluster.apps.kubeblocks.io/mysql-fphydk created apply test_create_mysql-fphydk.yaml Success `rm -rf test_create_mysql-fphydk.yaml` check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Creating May 28,2025 11:36 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check pod mysql-fphydk-mysql-0 container_name mysql exist password i4Y8Kow879 check pod mysql-fphydk-mysql-0 container_name mysql-exporter exist password i4Y8Kow879 check pod mysql-fphydk-mysql-0 container_name kbagent exist password i4Y8Kow879 check pod mysql-fphydk-mysql-0 container_name config-manager exist password i4Y8Kow879 No container logs contain secret password. describe cluster `kbcli cluster describe mysql-fphydk --namespace ns-cbpst ` Name: mysql-fphydk Created Time: May 28,2025 11:36 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-cbpst mysql semisync Running WipeOut Endpoints: COMPONENT INTERNAL EXTERNAL mysql mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql 8.0.30 mysql-fphydk-mysql-0 primary Running us-west-2a ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql 8.0.30 mysql-fphydk-mysql-1 secondary Running us-west-2a ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql 100m / 100m 512Mi / 512Mi data:10Gi kb-default-sc Images: COMPONENT COMPONENT-DEFINITION IMAGE mysql mysql-8.0-1.0.0-alpha.0 docker.io/apecloud/mysql:8.0.30 docker.io/apecloud/mysqld-exporter:0.15.1 docker.io/apecloud/kubeblocks-tools:1.0.0 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-cbpst mysql-fphydk `kbcli cluster label mysql-fphydk app.kubernetes.io/instance- --namespace ns-cbpst ` label "app.kubernetes.io/instance" not found. `kbcli cluster label mysql-fphydk app.kubernetes.io/instance=mysql-fphydk --namespace ns-cbpst ` `kbcli cluster label mysql-fphydk --list --namespace ns-cbpst ` NAME NAMESPACE LABELS mysql-fphydk ns-cbpst app.kubernetes.io/instance=mysql-fphydk clusterdefinition.kubeblocks.io/name=mysql label cluster app.kubernetes.io/instance=mysql-fphydk Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=mysql-fphydk --namespace ns-cbpst ` `kbcli cluster label mysql-fphydk --list --namespace ns-cbpst ` NAME NAMESPACE LABELS mysql-fphydk ns-cbpst app.kubernetes.io/instance=mysql-fphydk case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=mysql label cluster case.name=kbcli.test1 Success `kbcli cluster label mysql-fphydk case.name=kbcli.test2 --overwrite --namespace ns-cbpst ` `kbcli cluster label mysql-fphydk --list --namespace ns-cbpst ` NAME NAMESPACE LABELS mysql-fphydk ns-cbpst app.kubernetes.io/instance=mysql-fphydk case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=mysql label cluster case.name=kbcli.test2 Success `kbcli cluster label mysql-fphydk case.name- --namespace ns-cbpst ` `kbcli cluster label mysql-fphydk --list --namespace ns-cbpst ` NAME NAMESPACE LABELS mysql-fphydk ns-cbpst app.kubernetes.io/instance=mysql-fphydk clusterdefinition.kubeblocks.io/name=mysql delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: mysql-fphydk-mysql-0.mysql-fphydk-mysql-headless Master_User: kbreplicator Master_Port: 3306 Connect_Retry: 60 Master_Log_File: mysql-fphydk-mysql-0-bin.000004 Read_Master_Log_Pos: 39684 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 39890 Relay_Master_Log_File: mysql-fphydk-mysql-0-bin.000004 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 39684 Relay_Log_Space: 40094 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: Yes Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 260e6d87-3b75-11f0-9d5f-d67b54a03f3b Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 260e6d87-3b75-11f0-9d5f-d67b54a03f3b:51-152 Executed_Gtid_Set: 260e6d87-3b75-11f0-9d5f-d67b54a03f3b:1-152 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pi4Y8Kow879 -P3306 " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 83 connect cluster Success set max_connections to 83 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-mysql-fphydk" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mysql-fphydk --namespace ns-cbpst ` Error from server (NotFound): pods "test-db-client-executionloop-mysql-fphydk" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-mysql-fphydk" not found `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-mysql-fphydk namespace: ns-cbpst spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mysql-fphydk-mysql.ns-cbpst.svc.cluster.local" - "--user" - "root" - "--password" - "i4Y8Kow879" - "--port" - "3306" - "--dbtype" - "mysql" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-mysql-fphydk.yaml` pod/test-db-client-executionloop-mysql-fphydk created apply test-db-client-executionloop-mysql-fphydk.yaml Success `rm -rf test-db-client-executionloop-mysql-fphydk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 52s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 58s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 64s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 1/1 Running 0 70s check pod test-db-client-executionloop-mysql-fphydk status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mysql-fphydk 0/1 Completed 0 76s check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done --host mysql-fphydk-mysql.ns-cbpst.svc.cluster.local --user root --password i4Y8Kow879 --port 3306 --dbtype mysql --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 03:41:37.412 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255), tinyint_col TINYINT, smallint_col SMALLINT, mediumint_col MEDIUMINT, int_col INT, bigint_col BIGINT, float_col FLOAT, double_col DOUBLE, decimal_col DECIMAL(10, 2), date_col DATE, time_col TIME, datetime_col DATETIME, timestamp_col TIMESTAMP, year_col YEAR, char_col CHAR(10), text_col TEXT, blob_col BLOB, binary_col BINARY(10), varbinary_col VARBINARY(255), enum_col ENUM('Option1', 'Option2', 'Option3'), set_col SET('Value1', 'Value2', 'Value3') ); Execution loop start:INSERT INTO executions_loop.executions_loop_table (value, tinyint_col, smallint_col, mediumint_col, int_col, bigint_col, float_col, double_col, decimal_col, date_col, time_col, datetime_col, timestamp_col, year_col, char_col, text_col, blob_col, binary_col, varbinary_col, enum_col, set_col) VALUES ('executions_loop_test_1', 115, 15437, 2343116, 1176310342, -8790089858452577807, 0.9308553, 0.030179232146636203, 55.83809458056218, '2025-05-28', '03:41:38', '2025-05-28 03:41:38.211', CURRENT_TIMESTAMP, 2014, 'znPDQ8CoPw', '1ki5gdvsUM1Rkm6TJ696RBpPYEYcKGjvW80Vr3EWuquT0pMnfY4v7uTMxJKj0o0I04H3GGFSPOLVWY99HqSU56yF7FD5AjtnI5ABKg41cmC6UCb0QMOBMavcxhTZcjNO02OORiBTlcQGYIyioAPc6vDoRYrb28vJdADtmUNJyU8rCYk2BQVt2eSGvA6nso32pmbBV1lIOP9CcLPtAXq0PEL2MmKBKTacTm6ighX5CPnaigQFh4rOxf3vRw1c2AS', UNHEX(REPLACE('bedb9bdff124c89b0179', ' ', '')), UNHEX(REPLACE('45c87dd8ff31ad80a0de', ' ', '')), UNHEX(REPLACE('da5e64c9f1b7b63e63ec95676e23a0da080b0339a2688500a06a901878fbc9b5dd6171990ac4ad40b3fea1ea42b40915f96437e08648ede0042305900e468b6113d0165c92e0eef7c86e7e99713acd84f93defd97f3d6017a6ece13222e86d10d51951fa63a9f5229cb74443141870e0b1e0777b508484842402b1ecc7677c063c77e89fc7e02b2792ebf0cbdbfc259fce97789a6ef30507bfc766a3e65d8e23a99c5041014a5473ebbc2f8286842d73812ac9fa86fdc3b9dfdb3755b97c40203d7f1a65c45386a15216b5e06b25b2feff68eb2cf583cd678db7f0c750e676e661292aabba92044c51439dcdf46e8d7002c979884e7914d4f8de70420fd363', ' ', '')), 'Option3', 'Value3' ); [ 1s ] executions total: 12 successful: 12 failed: 0 disconnect: 0 [ 2s ] executions total: 34 successful: 34 failed: 0 disconnect: 0 [ 3s ] executions total: 60 successful: 60 failed: 0 disconnect: 0 [ 4s ] executions total: 84 successful: 84 failed: 0 disconnect: 0 [ 5s ] executions total: 108 successful: 108 failed: 0 disconnect: 0 [ 6s ] executions total: 131 successful: 131 failed: 0 disconnect: 0 [ 7s ] executions total: 154 successful: 154 failed: 0 disconnect: 0 [ 8s ] executions total: 176 successful: 176 failed: 0 disconnect: 0 [ 9s ] executions total: 202 successful: 202 failed: 0 disconnect: 0 [ 10s ] executions total: 228 successful: 228 failed: 0 disconnect: 0 [ 11s ] executions total: 250 successful: 250 failed: 0 disconnect: 0 [ 12s ] executions total: 274 successful: 274 failed: 0 disconnect: 0 [ 13s ] executions total: 299 successful: 299 failed: 0 disconnect: 0 [ 14s ] executions total: 323 successful: 323 failed: 0 disconnect: 0 [ 15s ] executions total: 346 successful: 346 failed: 0 disconnect: 0 [ 16s ] executions total: 367 successful: 367 failed: 0 disconnect: 0 [ 17s ] executions total: 389 successful: 389 failed: 0 disconnect: 0 [ 18s ] executions total: 413 successful: 413 failed: 0 disconnect: 0 [ 19s ] executions total: 431 successful: 431 failed: 0 disconnect: 0 [ 20s ] executions total: 452 successful: 452 failed: 0 disconnect: 0 [ 21s ] executions total: 473 successful: 473 failed: 0 disconnect: 0 [ 22s ] executions total: 494 successful: 494 failed: 0 disconnect: 0 [ 23s ] executions total: 512 successful: 512 failed: 0 disconnect: 0 [ 24s ] executions total: 533 successful: 533 failed: 0 disconnect: 0 [ 25s ] executions total: 553 successful: 553 failed: 0 disconnect: 0 [ 26s ] executions total: 575 successful: 575 failed: 0 disconnect: 0 [ 27s ] executions total: 597 successful: 597 failed: 0 disconnect: 0 [ 28s ] executions total: 621 successful: 621 failed: 0 disconnect: 0 [ 29s ] executions total: 643 successful: 643 failed: 0 disconnect: 0 [ 30s ] executions total: 664 successful: 664 failed: 0 disconnect: 0 [ 31s ] executions total: 683 successful: 683 failed: 0 disconnect: 0 [ 32s ] executions total: 704 successful: 704 failed: 0 disconnect: 0 [ 33s ] executions total: 725 successful: 725 failed: 0 disconnect: 0 [ 34s ] executions total: 746 successful: 746 failed: 0 disconnect: 0 [ 35s ] executions total: 768 successful: 768 failed: 0 disconnect: 0 [ 36s ] executions total: 791 successful: 791 failed: 0 disconnect: 0 [ 37s ] executions total: 812 successful: 812 failed: 0 disconnect: 0 [ 38s ] executions total: 833 successful: 833 failed: 0 disconnect: 0 [ 39s ] executions total: 855 successful: 855 failed: 0 disconnect: 0 [ 40s ] executions total: 876 successful: 876 failed: 0 disconnect: 0 [ 41s ] executions total: 901 successful: 901 failed: 0 disconnect: 0 [ 42s ] executions total: 926 successful: 926 failed: 0 disconnect: 0 [ 43s ] executions total: 950 successful: 950 failed: 0 disconnect: 0 [ 44s ] executions total: 977 successful: 977 failed: 0 disconnect: 0 [ 45s ] executions total: 1003 successful: 1003 failed: 0 disconnect: 0 [ 46s ] executions total: 1027 successful: 1027 failed: 0 disconnect: 0 [ 47s ] executions total: 1051 successful: 1051 failed: 0 disconnect: 0 [ 48s ] executions total: 1075 successful: 1075 failed: 0 disconnect: 0 [ 49s ] executions total: 1099 successful: 1099 failed: 0 disconnect: 0 [ 50s ] executions total: 1118 successful: 1118 failed: 0 disconnect: 0 [ 51s ] executions total: 1141 successful: 1141 failed: 0 disconnect: 0 [ 52s ] executions total: 1165 successful: 1165 failed: 0 disconnect: 0 [ 53s ] executions total: 1190 successful: 1190 failed: 0 disconnect: 0 [ 54s ] executions total: 1214 successful: 1214 failed: 0 disconnect: 0 [ 55s ] executions total: 1237 successful: 1237 failed: 0 disconnect: 0 [ 56s ] executions total: 1258 successful: 1258 failed: 0 disconnect: 0 [ 57s ] executions total: 1281 successful: 1281 failed: 0 disconnect: 0 [ 58s ] executions total: 1306 successful: 1306 failed: 0 disconnect: 0 [ 60s ] executions total: 1321 successful: 1321 failed: 0 disconnect: 0 Test Result: Total Executions: 1321 Successful Executions: 1321 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: mysql Host: mysql-fphydk-mysql.ns-cbpst.svc.cluster.local Port: 3306 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 1321 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mysql-fphydk --namespace ns-cbpst ` pod/test-db-client-executionloop-mysql-fphydk patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-mysql-fphydk" force deleted LB_TYPE is set to: internet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose mysql-fphydk --auto-approve --force=true --type internet --enable true --components mysql --role-selector primary --namespace ns-cbpst ` OpsRequest mysql-fphydk-expose-sh6sw created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-expose-sh6sw -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-expose-sh6sw ns-cbpst Expose mysql-fphydk mysql Running 0/1 May 28,2025 11:43 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-expose-sh6sw ns-cbpst Expose mysql-fphydk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 check ops status done ops_status:mysql-fphydk-expose-sh6sw ns-cbpst Expose mysql-fphydk mysql Succeed 1/1 May 28,2025 11:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-expose-sh6sw --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-expose-sh6sw patched `kbcli cluster delete-ops --name mysql-fphydk-expose-sh6sw --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-expose-sh6sw deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-fphydk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-mysql-fphydk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mysql-fphydk created apply test-chaos-mesh-networkcorruptover-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-mysql-fphydk.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:10Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1;secondary: No resources found in ns-cbpst namespace. primary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1;secondary: check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-fphydk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=mysql-fphydk,apps.kubeblocks.io/component-name=mysql,apps.kubeblocks.io/vct-name=data --namespace ns-cbpst ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in mysql-fphydk namespace. `kbcli cluster volume-expand mysql-fphydk --auto-approve --force=true --components mysql --volume-claim-templates data --storage 14Gi --namespace ns-cbpst ` OpsRequest mysql-fphydk-volumeexpansion-vn9vg created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-volumeexpansion-vn9vg -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-volumeexpansion-vn9vg ns-cbpst VolumeExpansion mysql-fphydk mysql Running 0/2 May 28,2025 11:46 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done No resources found in mysql-fphydk namespace. check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-volumeexpansion-vn9vg ns-cbpst VolumeExpansion mysql-fphydk mysql Succeed 2/2 May 28,2025 11:46 UTC+0800 check ops status done ops_status:mysql-fphydk-volumeexpansion-vn9vg ns-cbpst VolumeExpansion mysql-fphydk mysql Succeed 2/2 May 28,2025 11:46 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-volumeexpansion-vn9vg --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-volumeexpansion-vn9vg patched `kbcli cluster delete-ops --name mysql-fphydk-volumeexpansion-vn9vg --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-volumeexpansion-vn9vg deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-fphydk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-mysql-fphydk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-mysql-fphydk created apply test-chaos-mesh-dnsrandom-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-dnsrandom-mysql-fphydk.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:37 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-fphydk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover delete pod:mysql-fphydk-mysql-1 `kubectl delete pod mysql-fphydk-mysql-1 --namespace ns-cbpst ` pod "mysql-fphydk-mysql-1" deleted check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check failover pod name failover pod name:mysql-fphydk-mysql-0 failover Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster configure component_tmp: mysql apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-reconfiguring- namespace: ns-cbpst spec: type: Reconfiguring clusterName: mysql-fphydk force: true reconfigures: - componentName: mysql parameters: - key: binlog_expire_logs_seconds value: '691200' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-reconfiguring-2j5ws created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-reconfiguring-2j5ws ns-cbpst Reconfiguring mysql-fphydk mysql,mysql Running -/- May 28,2025 11:52 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-reconfiguring-2j5ws ns-cbpst Reconfiguring mysql-fphydk mysql,mysql Succeed -/- May 28,2025 11:52 UTC+0800 check ops status done ops_status:mysql-fphydk-reconfiguring-2j5ws ns-cbpst Reconfiguring mysql-fphydk mysql,mysql Succeed -/- May 28,2025 11:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-reconfiguring-2j5ws --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-reconfiguring-2j5ws patched `kbcli cluster delete-ops --name mysql-fphydk-reconfiguring-2j5ws --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-reconfiguring-2j5ws deleted component_config:mysql check config variables `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'binlog_expire_logs_seconds'\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash | (grep "binlog_expire_logs_seconds" || true) | awk '***print mysql***' current value_actual: 691200 configure:[binlog_expire_logs_seconds] result actual:[691200] equal expected:[691200] `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-fphydk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-mysql-fphydk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-mysql-fphydk created apply test-chaos-mesh-networkbandwidthover-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-mysql-fphydk.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-fphydk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-fphydk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-mysql-fphydk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-mysql-fphydk created apply test-chaos-mesh-fullcpu-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-fullcpu-mysql-fphydk.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-fphydk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-mysql-fphydk" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-mysql-fphydk --namespace ns-cbpst ` Error from server (NotFound): pods "test-db-client-connectionstress-mysql-fphydk" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-mysql-fphydk" not found `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-mysql-fphydk namespace: ns-cbpst spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mysql-fphydk-mysql.ns-cbpst.svc.cluster.local" - "--user" - "root" - "--password" - "i4Y8Kow879" - "--port" - "3306" - "--database" - "mysql" - "--dbtype" - "mysql" - "--test" - "connectionstress" - "--connections" - "83" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-mysql-fphydk.yaml` pod/test-db-client-connectionstress-mysql-fphydk created apply test-db-client-connectionstress-mysql-fphydk.yaml Success `rm -rf test-db-client-connectionstress-mysql-fphydk.yaml` check pod status check pod test-db-client-connectionstress-mysql-fphydk status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-mysql-fphydk 0/1 Completed 0 6s check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done --host mysql-fphydk-mysql.ns-cbpst.svc.cluster.local --user root --password i4Y8Kow879 --port 3306 --database mysql --dbtype mysql --test connectionstress --connections 83 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 03:58:53.528 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered 03:58:55.694 [main] DEBUG com.yashandb.log.LoggerFactory -- Logger type is SLF4J Logger. 03:58:55.697 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306/mysql?useSSL=false&allowPublicKeyRetrieval=true Failed to connect to MySQL database: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" Trying with database MySQL. 03:58:55.704 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306?useSSL=false&allowPublicKeyRetrieval=true java.io.IOException: Failed to connect to MySQL database: at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:63) at com.apecloud.dbtester.tester.MySQLTester.connectionStress(MySQLTester.java:111) at com.apecloud.dbtester.tester.TestExecutor.executeTest(TestExecutor.java:34) at OneClient.executeTest(OneClient.java:105) at OneClient.main(OneClient.java:37) Caused by: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:61) ... 4 more 03:58:55.713 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306/mysql?useSSL=false&allowPublicKeyRetrieval=true Failed to connect to MySQL database: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" Trying with database MySQL. 03:58:55.719 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306?useSSL=false&allowPublicKeyRetrieval=true java.io.IOException: Failed to connect to MySQL database: at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:63) at com.apecloud.dbtester.tester.MySQLTester.connectionStress(MySQLTester.java:111) at com.apecloud.dbtester.tester.TestExecutor.executeTest(TestExecutor.java:34) at OneClient.executeTest(OneClient.java:105) at OneClient.main(OneClient.java:37) Caused by: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:61) ... 4 more 03:58:55.724 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306/mysql?useSSL=false&allowPublicKeyRetrieval=true Failed to connect to MySQL database: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" Trying with database MySQL. java.io.IOException: Failed to connect to MySQL database: 03:58:55.727 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://mysql-fphydk-mysql.ns-cbpst.svc.cluster.local:3306?useSSL=false&allowPublicKeyRetrieval=true at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:63) at com.apecloud.dbtester.tester.MySQLTester.connectionStress(MySQLTester.java:111) at com.apecloud.dbtester.tester.TestExecutor.executeTest(TestExecutor.java:34) at OneClient.executeTest(OneClient.java:105) at OneClient.main(OneClient.java:37) Caused by: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:61) ... 4 more Test Result: null Connection Information: Database Type: mysql Host: mysql-fphydk-mysql.ns-cbpst.svc.cluster.local Port: 3306 Database: mysql Table: User: root Org: Access Mode: mysql Test Type: connectionstress Connection Count: 83 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-mysql-fphydk --namespace ns-cbpst ` pod/test-db-client-connectionstress-mysql-fphydk patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-mysql-fphydk" force deleted check failover pod name failover pod name:mysql-fphydk-mysql-1 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-fphydk" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-mysql-fphydk.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-mysql-fphydk created apply test-chaos-mesh-dnserror-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-dnserror-mysql-fphydk.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mysql-fphydk --namespace ns-cbpst ` dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-fphydk" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster mysql scale-out check cluster status before ops check cluster status done cluster_status:Running No resources found in mysql-fphydk namespace. `kbcli cluster scale-out mysql-fphydk --auto-approve --force=true --components mysql --replicas 1 --namespace ns-cbpst ` OpsRequest mysql-fphydk-horizontalscaling-46np8 created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-horizontalscaling-46np8 -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-horizontalscaling-46np8 ns-cbpst HorizontalScaling mysql-fphydk mysql Running 0/1 May 28,2025 12:02 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 mysql-fphydk-mysql-2 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:02 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done No resources found in mysql-fphydk namespace. check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-horizontalscaling-46np8 ns-cbpst HorizontalScaling mysql-fphydk mysql Succeed 1/1 May 28,2025 12:02 UTC+0800 check ops status done ops_status:mysql-fphydk-horizontalscaling-46np8 ns-cbpst HorizontalScaling mysql-fphydk mysql Succeed 1/1 May 28,2025 12:02 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-horizontalscaling-46np8 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-horizontalscaling-46np8 patched `kbcli cluster delete-ops --name mysql-fphydk-horizontalscaling-46np8 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-horizontalscaling-46np8 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster mysql scale-in check cluster status before ops check cluster status done cluster_status:Running No resources found in mysql-fphydk namespace. `kbcli cluster scale-in mysql-fphydk --auto-approve --force=true --components mysql --replicas 1 --namespace ns-cbpst ` OpsRequest mysql-fphydk-horizontalscaling-rcccv created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-horizontalscaling-rcccv -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-horizontalscaling-rcccv ns-cbpst HorizontalScaling mysql-fphydk mysql Running 0/1 May 28,2025 12:03 UTC+0800 ops_status:mysql-fphydk-horizontalscaling-rcccv ns-cbpst HorizontalScaling mysql-fphydk mysql Succeed 1/1 May 28,2025 12:03 UTC+0800 ops HorizontalScaling Succeed or Failed Soon check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done No resources found in mysql-fphydk namespace. check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-horizontalscaling-rcccv ns-cbpst HorizontalScaling mysql-fphydk mysql Succeed 1/1 May 28,2025 12:03 UTC+0800 check ops status done ops_status:mysql-fphydk-horizontalscaling-rcccv ns-cbpst HorizontalScaling mysql-fphydk mysql Succeed 1/1 May 28,2025 12:03 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-horizontalscaling-rcccv --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-horizontalscaling-rcccv patched `kbcli cluster delete-ops --name mysql-fphydk-horizontalscaling-rcccv --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-horizontalscaling-rcccv deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster does not need to check monitor currently check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 11:36 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:14Gi ip-172-31-13-128.us-west-2.compute.internal/172.31.13.128 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale mysql-fphydk --auto-approve --force=true --components mysql --cpu 200m --memory 0.6Gi --namespace ns-cbpst ` OpsRequest mysql-fphydk-verticalscaling-nmtl5 created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-verticalscaling-nmtl5 -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-verticalscaling-nmtl5 ns-cbpst VerticalScaling mysql-fphydk mysql Running 0/2 May 28,2025 12:04 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:04 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:05 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-verticalscaling-nmtl5 ns-cbpst VerticalScaling mysql-fphydk mysql Succeed 2/2 May 28,2025 12:04 UTC+0800 check ops status done ops_status:mysql-fphydk-verticalscaling-nmtl5 ns-cbpst VerticalScaling mysql-fphydk mysql Succeed 2/2 May 28,2025 12:04 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-verticalscaling-nmtl5 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-verticalscaling-nmtl5 patched `kbcli cluster delete-ops --name mysql-fphydk-verticalscaling-nmtl5 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-verticalscaling-nmtl5 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-fphydk" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-mysql-fphydk.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-mysql-fphydk created apply test-chaos-mesh-timeoffset-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-timeoffset-mysql-fphydk.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:04 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:05 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-fphydk" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-fphydk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 action: partition mode: all target: mode: all selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-mysql-fphydk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-mysql-fphydk created apply test-chaos-mesh-networkpartition-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-networkpartition-mysql-fphydk.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:04 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:05 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-fphydk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop mysql-fphydk --auto-approve --force=true --namespace ns-cbpst ` OpsRequest mysql-fphydk-stop-fdj2h created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-stop-fdj2h -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-stop-fdj2h ns-cbpst Stop mysql-fphydk mysql Running 0/2 May 28,2025 12:12 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Stopped May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-stop-fdj2h ns-cbpst Stop mysql-fphydk mysql Succeed 2/2 May 28,2025 12:12 UTC+0800 check ops status done ops_status:mysql-fphydk-stop-fdj2h ns-cbpst Stop mysql-fphydk mysql Succeed 2/2 May 28,2025 12:12 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-stop-fdj2h --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-stop-fdj2h patched `kbcli cluster delete-ops --name mysql-fphydk-stop-fdj2h --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-stop-fdj2h deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start mysql-fphydk --force=true --namespace ns-cbpst ` OpsRequest mysql-fphydk-start-j9qxn created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-start-j9qxn -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-start-j9qxn ns-cbpst Start mysql-fphydk mysql Running 0/2 May 28,2025 12:13 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 12:13 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:13 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-start-j9qxn ns-cbpst Start mysql-fphydk mysql Succeed 2/2 May 28,2025 12:13 UTC+0800 check ops status done ops_status:mysql-fphydk-start-j9qxn ns-cbpst Start mysql-fphydk mysql Succeed 2/2 May 28,2025 12:13 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-start-j9qxn --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-start-j9qxn patched `kbcli cluster delete-ops --name mysql-fphydk-start-j9qxn --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-start-j9qxn deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-226.us-west-2.compute.internal/172.31.13.226 May 28,2025 12:13 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:13 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check failover pod name failover pod name:mysql-fphydk-mysql-0 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test switchover apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-switchover- namespace: ns-cbpst spec: clusterName: mysql-fphydk type: Switchover switchover: - componentName: mysql candidateName: 'mysql-fphydk-mysql-1' instanceName: 'mysql-fphydk-mysql-0' `kubectl create -f test_create_mysql-fphydk_switchover.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-switchover-nwbml created create test_create_mysql-fphydk_switchover.yaml Success `rm -rf test_create_mysql-fphydk_switchover.yaml` switchover pod:mysql-fphydk-mysql-0 checking switchover... switchover pod:mysql-fphydk-mysql-1 switchover success No resources found in ns-cbpst namespace. `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-mysql-fphydk namespace: ns-cbpst spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:mysql-fphydk-mysql.ns-cbpst.svc.cluster.local,user:root,password:i4Y8Kow879,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-mysql-fphydk.yaml` pod/benchtest-mysql-fphydk created apply benchtest-mysql-fphydk.yaml Success `rm -rf benchtest-mysql-fphydk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 24s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 30s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 36s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 42s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 48s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 ContainerCreating 0 54s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 1/1 Running 0 60s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 1/1 Running 0 66s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 1/1 Running 0 72s check pod benchtest-mysql-fphydk status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 Completed 0 78s `kubectl logs benchtest-mysql-fphydk --tail 30 --namespace ns-cbpst ` [ 7s ] thds: 4 tps: 20.00 qps: 401.04 (r/w/o: 275.03/86.01/40.00) lat (ms,99%): 223.34 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 25.00 qps: 498.98 (r/w/o: 348.98/100.00/50.00) lat (ms,99%): 297.92 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 26.00 qps: 519.98 (r/w/o: 360.99/107.00/52.00) lat (ms,99%): 287.38 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 24.00 qps: 453.05 (r/w/o: 320.03/85.01/48.00) lat (ms,99%): 292.60 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 3220 write: 920 other: 460 total: 4600 transactions: 230 (22.47 per sec.) queries: 4600 (449.48 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.2318s total number of events: 230 Latency (ms): min: 14.77 avg: 176.53 max: 312.67 99th percentile: 297.92 sum: 40602.97 Threads fairness: events (avg/stddev): 57.5000/1.66 execution time (avg/stddev): 10.1507/0.06 `kubectl delete pod benchtest-mysql-fphydk --force --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-mysql-fphydk" force deleted LB_TYPE is set to: internet No resources found in ns-cbpst namespace. `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-mysql-fphydk namespace: ns-cbpst spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:ae49d6b9e065c4bd29e606fbb9c87d14-732ba00b4cae2c76.elb.us-west-2.amazonaws.com,user:root,password:i4Y8Kow879,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-mysql-fphydk.yaml` pod/benchtest-mysql-fphydk created apply benchtest-mysql-fphydk.yaml Success `rm -rf benchtest-mysql-fphydk.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 1/1 Running 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 1/1 Running 0 12s check pod benchtest-mysql-fphydk status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-mysql-fphydk 0/1 Completed 0 18s `kubectl logs benchtest-mysql-fphydk --tail 30 --namespace ns-cbpst ` [ 7s ] thds: 4 tps: 19.00 qps: 393.02 (r/w/o: 272.02/83.00/38.00) lat (ms,99%): 397.39 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 22.00 qps: 407.98 (r/w/o: 283.99/80.00/44.00) lat (ms,99%): 297.92 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 20.00 qps: 427.00 (r/w/o: 299.00/88.00/40.00) lat (ms,99%): 287.38 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 21.00 qps: 427.99 (r/w/o: 302.99/83.00/42.00) lat (ms,99%): 292.60 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 3052 write: 872 other: 436 total: 4360 transactions: 218 (21.58 per sec.) queries: 4360 (431.55 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.1015s total number of events: 218 Latency (ms): min: 98.13 avg: 185.25 max: 394.67 99th percentile: 297.92 sum: 40385.02 Threads fairness: events (avg/stddev): 54.5000/2.06 execution time (avg/stddev): 10.0963/0.00 `kubectl delete pod benchtest-mysql-fphydk --force --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-mysql-fphydk" force deleted test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mysql-fphydk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-mysql-fphydk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-mysql-fphydk created apply test-chaos-mesh-networklossover-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-networklossover-mysql-fphydk.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:16 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:13 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mysql-fphydk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart mysql-fphydk --auto-approve --force=true --namespace ns-cbpst ` OpsRequest mysql-fphydk-restart-m675r created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-restart-m675r -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-restart-m675r ns-cbpst Restart mysql-fphydk mysql Running 0/2 May 28,2025 12:23 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-restart-m675r ns-cbpst Restart mysql-fphydk mysql Succeed 2/2 May 28,2025 12:23 UTC+0800 check ops status done ops_status:mysql-fphydk-restart-m675r ns-cbpst Restart mysql-fphydk mysql Succeed 2/2 May 28,2025 12:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-restart-m675r --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-restart-m675r patched `kbcli cluster delete-ops --name mysql-fphydk-restart-m675r --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-restart-m675r deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-fphydk" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mysql-fphydk.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mysql-fphydk created apply test-chaos-mesh-oom-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-oom-mysql-fphydk.yaml` check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-fphydk" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-fphydk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-mysql-fphydk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-mysql-fphydk created apply test-chaos-mesh-networkduplicate-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-networkduplicate-mysql-fphydk.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-fphydk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cmpv upgrade service version:2,5.7.44|2,8.0.30|2,8.0.31|2,8.0.32|2,8.0.33|2,8.0.34|2,8.0.35|2,8.0.36|2,8.0.37|2,8.0.38|2,8.0.39|2,8.4.0|2,8.4.1|2,8.4.2 set latest cmpv service version latest service version:8.0.39 cmpv service version upgrade and downgrade 8.0.30 is less than 8.0.31 upgrade from:8.0.30 to service version:8.0.31 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.31 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-6779c created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-6779c ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:28 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-6779c ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:28 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-6779c ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-6779c --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-6779c patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-6779c --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-6779c deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success 8.0.31 is less than 8.0.32 upgrade from:8.0.31 to service version:8.0.32 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.32 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-s6nmj created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-s6nmj ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:30 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-s6nmj ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:30 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-s6nmj ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-s6nmj --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-s6nmj patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-s6nmj --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-s6nmj deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success 8.0.32 is less than 8.0.33 upgrade from:8.0.32 to service version:8.0.33 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.33 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-d2lgh created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-d2lgh ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:32 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` connect checking... connect checking... connect checking... check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-d2lgh ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:32 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-d2lgh ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:32 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-d2lgh --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-d2lgh patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-d2lgh --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-d2lgh deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success 8.0.33 is less than 8.0.34 upgrade from:8.0.33 to service version:8.0.34 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.34 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-wc8w9 created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-wc8w9 ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:34 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-wc8w9 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:34 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-wc8w9 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:34 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-wc8w9 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-wc8w9 patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-wc8w9 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-wc8w9 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.34 to service version:8.0.35 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.35 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-vldg4 created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-vldg4 ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:36 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 No resources found in ns-cbpst namespace. mysql-fphydk-mysql-1 primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-vldg4 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:36 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-vldg4 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:36 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-vldg4 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-vldg4 patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-vldg4 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-vldg4 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.35 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-nvjp7 created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-nvjp7 ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:39 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-nvjp7 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:39 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-nvjp7 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:39 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-nvjp7 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-nvjp7 patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-nvjp7 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-nvjp7 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.35 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.35 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-l6d8b created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-l6d8b ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:41 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-l6d8b ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:41 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-l6d8b ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-l6d8b --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-l6d8b patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-l6d8b --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-l6d8b deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.35 to service version:8.0.36 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.36 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-8pcxh created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-8pcxh ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:42 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-8pcxh ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:42 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-8pcxh ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:42 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-8pcxh --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-8pcxh patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-8pcxh --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-8pcxh deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.36 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-cd5gr created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-cd5gr ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:44 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-cd5gr ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:44 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-cd5gr ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-cd5gr --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-cd5gr patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-cd5gr --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-cd5gr deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.36 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.36 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-l6whc created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-l6whc ns-cbpst Upgrade mysql-fphydk Running -/- May 28,2025 12:46 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-l6whc ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:46 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-l6whc ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:46 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-l6whc --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-l6whc patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-l6whc --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-l6whc deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.36 to service version:8.0.37 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.37 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-5df5w created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-5df5w ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:47 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 No resources found in ns-cbpst namespace. mysql-fphydk-mysql-1 primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-5df5w ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:47 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-5df5w ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-5df5w --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-5df5w patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-5df5w --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-5df5w deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.37 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-hmww2 created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-hmww2 ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:49 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 No resources found in ns-cbpst namespace. mysql-fphydk-mysql-1 primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-hmww2 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:49 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-hmww2 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:49 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-hmww2 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-hmww2 patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-hmww2 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-hmww2 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.37 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.37 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-g6sn5 created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-g6sn5 ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:51 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-g6sn5 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:51 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-g6sn5 ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:51 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-g6sn5 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-g6sn5 patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-g6sn5 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-g6sn5 deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.37 to service version:8.0.38 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.38 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-kkqkw created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-kkqkw ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:52 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-kkqkw ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:52 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-kkqkw ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-kkqkw --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-kkqkw patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-kkqkw --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-kkqkw deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success upgrade from:8.0.38 to service version:8.0.39 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.39 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-sszzz created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-sszzz ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:54 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 No resources found in ns-cbpst namespace. primary: ;secondary: mysql-fphydk-mysql-0 mysql-fphydk-mysql-1 check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-sszzz ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:54 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-sszzz ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:54 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-sszzz --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-sszzz patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-sszzz --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-sszzz deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success downgrade from:8.0.39 to service version:8.0.38 cluster upgrade apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-upgrade-cmpv- namespace: ns-cbpst spec: clusterName: mysql-fphydk upgrade: components: - componentName: mysql serviceVersion: 8.0.38 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-rnfxz created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-rnfxz ns-cbpst Upgrade mysql-fphydk mysql Running 0/2 May 28,2025 12:56 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-upgrade-cmpv-rnfxz ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:56 UTC+0800 check ops status done ops_status:mysql-fphydk-upgrade-cmpv-rnfxz ns-cbpst Upgrade mysql-fphydk mysql Succeed 2/2 May 28,2025 12:56 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-upgrade-cmpv-rnfxz --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-upgrade-cmpv-rnfxz patched `kbcli cluster delete-ops --name mysql-fphydk-upgrade-cmpv-rnfxz --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-upgrade-cmpv-rnfxz deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-fphydk" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-mysql-fphydk.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-mysql-fphydk created apply test-chaos-mesh-podfailure-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-podfailure-mysql-fphydk.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Failed May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql cluster_status:Failed cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-0;secondary: mysql-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-fphydk" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod mysql-fphydk-mysql-0 --namespace ns-cbpst -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-9-182.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod mysql-fphydk-mysql-1 --namespace ns-cbpst -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-13-93.us-west-2.compute.internal success kubectl drain ip-172-31-9-182.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-9-182.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-7lx7h, kube-system/aws-node-pkd6m, kube-system/ebs-csi-node-ppc96, kube-system/kube-proxy-n4dxb evicting pod ns-dyjue/orcm-jjolcb-orchestrator-0 evicting pod ns-cbpst/mysql-fphydk-mysql-0 pod/orcm-jjolcb-orchestrator-0 evicted pod/mysql-fphydk-mysql-0 evicted node/ip-172-31-9-182.us-west-2.compute.internal drained kubectl uncordon ip-172-31-9-182.us-west-2.compute.internal node/ip-172-31-9-182.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 13:00 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done check failover pod name failover pod name:mysql-fphydk-mysql-1 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-fphydk" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-fphydk" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-mysql-fphydk namespace: ns-cbpst spec: selector: namespaces: - ns-cbpst labelSelectors: apps.kubeblocks.io/pod-name: mysql-fphydk-mysql-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-mysql-fphydk.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-mysql-fphydk created apply test-chaos-mesh-networkdelay-mysql-fphydk.yaml Success `rm -rf test-chaos-mesh-networkdelay-mysql-fphydk.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 13:00 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mysql-fphydk --namespace ns-cbpst ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-fphydk" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mysql-fphydk" not found check failover pod name failover pod name:mysql-fphydk-mysql-1 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update mysql-fphydk --termination-policy=WipeOut --namespace ns-cbpst ` cluster.apps.kubeblocks.io/mysql-fphydk updated (no change) check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-0 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 13:00 UTC+0800 mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-0 --namespace ns-cbpst -- bash ` check cluster connect done `kubectl get backupschedule -l app.kubernetes.io/instance=mysql-fphydk ` `kubectl get backupschedule mysql-fphydk-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched check backup status `kbcli cluster list-backups mysql-fphydk --namespace ns-cbpst ` No backups found in ns-cbpst namespace. No backups found in ns-cbpst namespace. backup_status:-- No backups found in ns-cbpst namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=mysql-fphydk ` `kubectl get backupschedule mysql-fphydk-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backup_status:mysql-fphydk-xtrabackup-Running backup_status:mysql-fphydk-xtrabackup-Running check backup status done backup_status:mysql-fphydk-xtrabackup-20250528050500 ns-cbpst mysql-fphydk xtrabackup Completed 6658756 20s Delete May 28,2025 13:05 UTC+0800 May 28,2025 13:05 UTC+0800 Jun 05,2025 13:05 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=mysql-fphydk ` `kubectl get backupschedule mysql-fphydk-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mysql-fphydk-mysql-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "abk-fphydk" not found `kbcli cluster describe-backup --names mysql-fphydk-xtrabackup-20250528050500 --namespace ns-cbpst ` Name: mysql-fphydk-xtrabackup-20250528050500 Cluster: mysql-fphydk Namespace: ns-cbpst Spec: Method: xtrabackup Policy Name: mysql-fphydk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-mysql-fphydk-xtrabackup-20250528050500-721791db TargetPodName: mysql-fphydk-mysql-0 Phase: Completed Start Time: May 28,2025 13:05 UTC+0800 Completion Time: May 28,2025 13:05 UTC+0800 Status: Phase: Completed Total Size: 6658756 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 20s Expiration Time: Jun 05,2025 13:05 UTC+0800 Start Time: May 28,2025 13:05 UTC+0800 Completion Time: May 28,2025 13:05 UTC+0800 Path: /ns-cbpst/mysql-fphydk-06b06dde-504d-44aa-b50f-1f5817ef30b3/mysql/mysql-fphydk-xtrabackup-20250528050500 Warning Events: `kbcli cluster restore abk-fphydk --backup mysql-fphydk-xtrabackup-20250528050500 --namespace ns-cbpst ` Cluster abk-fphydk created check cluster status `kbcli cluster list abk-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS abk-fphydk ns-cbpst mysql WipeOut Creating May 28,2025 13:05 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances abk-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME abk-fphydk-mysql-0 ns-cbpst abk-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 13:06 UTC+0800 abk-fphydk-mysql-1 ns-cbpst abk-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:07 UTC+0800 check pod status done check cluster role check cluster role done primary: abk-fphydk-mysql-0;secondary: abk-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it abk-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done `kbcli cluster describe-backup --names mysql-fphydk-xtrabackup-20250528050500 --namespace ns-cbpst ` Name: mysql-fphydk-xtrabackup-20250528050500 Cluster: mysql-fphydk Namespace: ns-cbpst Spec: Method: xtrabackup Policy Name: mysql-fphydk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-mysql-fphydk-xtrabackup-20250528050500-721791db TargetPodName: mysql-fphydk-mysql-0 Phase: Completed Start Time: May 28,2025 13:05 UTC+0800 Completion Time: May 28,2025 13:05 UTC+0800 Status: Phase: Completed Total Size: 6658756 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 20s Expiration Time: Jun 05,2025 13:05 UTC+0800 Start Time: May 28,2025 13:05 UTC+0800 Completion Time: May 28,2025 13:05 UTC+0800 Path: /ns-cbpst/mysql-fphydk-06b06dde-504d-44aa-b50f-1f5817ef30b3/mysql/mysql-fphydk-xtrabackup-20250528050500 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: abk-fphydk-mysql-0.abk-fphydk-mysql-headless Master_User: kbreplicator Master_Port: 3306 Connect_Retry: 60 Master_Log_File: abk-fphydk-mysql-0-bin.000001 Read_Master_Log_Pos: 18603 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 18845 Relay_Master_Log_File: abk-fphydk-mysql-0-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 18603 Relay_Log_Space: 19049 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: Yes Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: a3a23ed4-3b81-11f0-a63f-a2d335daa6ab Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: a3a23ed4-3b81-11f0-a63f-a2d335daa6ab:1-46 Executed_Gtid_Set: 260e6d87-3b75-11f0-9d5f-d67b54a03f3b:1-2427, 7cc22dd3-3b75-11f0-b535-ce095d4ba1c7:1-3155, a3a23ed4-3b81-11f0-a63f-a2d335daa6ab:1-46 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pi4Y8Kow879 -P3306 " | kubectl exec -it abk-fphydk-mysql-1 --namespace ns-cbpst -- bash ` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster abk-fphydk `kbcli cluster delete abk-fphydk --auto-approve --namespace ns-cbpst ` Cluster abk-fphydk deleted pod_info:abk-fphydk-mysql-0 4/4 Terminating 0 2m19s abk-fphydk-mysql-1 4/4 Terminating 0 54s No resources found in ns-cbpst namespace. delete cluster pod done No resources found in ns-cbpst namespace. check cluster resource non-exist OK: pvc No resources found in ns-cbpst namespace. delete cluster done No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups mysql-fphydk-xtrabackup-20250528050500 --namespace ns-cbpst ` backup.dataprotection.kubeblocks.io/mysql-fphydk-xtrabackup-20250528050500 patched `kbcli cluster delete-backup mysql-fphydk --name mysql-fphydk-xtrabackup-20250528050500 --force --auto-approve --namespace ns-cbpst ` Backup mysql-fphydk-xtrabackup-20250528050500 deleted cluster xtrabackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mysql-fphydk --method xtrabackup --namespace ns-cbpst ` Backup backup-ns-cbpst-mysql-fphydk-20250528130909 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-cbpst-mysql-fphydk-20250528130909 -n ns-cbpst check backup status `kbcli cluster list-backups mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-cbpst-mysql-fphydk-20250528130909 ns-cbpst mysql-fphydk xtrabackup Running Delete May 28,2025 13:09 UTC+0800 backup_status:mysql-fphydk-xtrabackup-Running backup_status:mysql-fphydk-xtrabackup-Running check backup status done backup_status:backup-ns-cbpst-mysql-fphydk-20250528130909 ns-cbpst mysql-fphydk xtrabackup Completed 6749691 11s Delete May 28,2025 13:09 UTC+0800 May 28,2025 13:09 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "abk-fphydk" not found `kbcli cluster describe-backup --names backup-ns-cbpst-mysql-fphydk-20250528130909 --namespace ns-cbpst ` Name: backup-ns-cbpst-mysql-fphydk-20250528130909 Cluster: mysql-fphydk Namespace: ns-cbpst Spec: Method: xtrabackup Policy Name: mysql-fphydk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-cbpst-mysql-fphydk-20250528130909-2753ceb TargetPodName: mysql-fphydk-mysql-0 Phase: Completed Start Time: May 28,2025 13:09 UTC+0800 Completion Time: May 28,2025 13:09 UTC+0800 Status: Phase: Completed Total Size: 6749691 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: May 28,2025 13:09 UTC+0800 Completion Time: May 28,2025 13:09 UTC+0800 Path: /ns-cbpst/mysql-fphydk-06b06dde-504d-44aa-b50f-1f5817ef30b3/mysql/backup-ns-cbpst-mysql-fphydk-20250528130909 Warning Events: `kbcli cluster restore abk-fphydk --backup backup-ns-cbpst-mysql-fphydk-20250528130909 --namespace ns-cbpst ` Cluster abk-fphydk created check cluster status `kbcli cluster list abk-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS abk-fphydk ns-cbpst mysql WipeOut Creating May 28,2025 13:09 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances abk-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME abk-fphydk-mysql-0 ns-cbpst abk-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 13:09 UTC+0800 abk-fphydk-mysql-1 ns-cbpst abk-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:10 UTC+0800 check pod status done check cluster role check cluster role done primary: abk-fphydk-mysql-0;secondary: abk-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it abk-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done `kbcli cluster describe-backup --names backup-ns-cbpst-mysql-fphydk-20250528130909 --namespace ns-cbpst ` Name: backup-ns-cbpst-mysql-fphydk-20250528130909 Cluster: mysql-fphydk Namespace: ns-cbpst Spec: Method: xtrabackup Policy Name: mysql-fphydk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-cbpst-mysql-fphydk-20250528130909-2753ceb TargetPodName: mysql-fphydk-mysql-0 Phase: Completed Start Time: May 28,2025 13:09 UTC+0800 Completion Time: May 28,2025 13:09 UTC+0800 Status: Phase: Completed Total Size: 6749691 ActionSet Name: mysql-xtrabackup-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: May 28,2025 13:09 UTC+0800 Completion Time: May 28,2025 13:09 UTC+0800 Path: /ns-cbpst/mysql-fphydk-06b06dde-504d-44aa-b50f-1f5817ef30b3/mysql/backup-ns-cbpst-mysql-fphydk-20250528130909 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: abk-fphydk-mysql-0.abk-fphydk-mysql-headless Master_User: kbreplicator Master_Port: 3306 Connect_Retry: 60 Master_Log_File: abk-fphydk-mysql-0-bin.000001 Read_Master_Log_Pos: 16197 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 16439 Relay_Master_Log_File: abk-fphydk-mysql-0-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 16197 Relay_Log_Space: 16643 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: Yes Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: 09cd8264-3b82-11f0-a715-4a1b7ca236a8 Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: 09cd8264-3b82-11f0-a715-4a1b7ca236a8:1-40 Executed_Gtid_Set: 09cd8264-3b82-11f0-a715-4a1b7ca236a8:1-40, 260e6d87-3b75-11f0-9d5f-d67b54a03f3b:1-2427, 7cc22dd3-3b75-11f0-b535-ce095d4ba1c7:1-3395 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pi4Y8Kow879 -P3306 " | kubectl exec -it abk-fphydk-mysql-1 --namespace ns-cbpst -- bash ` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster abk-fphydk `kbcli cluster delete abk-fphydk --auto-approve --namespace ns-cbpst ` Cluster abk-fphydk deleted pod_info:abk-fphydk-mysql-0 4/4 Terminating 0 78s abk-fphydk-mysql-1 4/4 Terminating 0 56s No resources found in ns-cbpst namespace. delete cluster pod done No resources found in ns-cbpst namespace. check cluster resource non-exist OK: pvc No resources found in ns-cbpst namespace. delete cluster done No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mysql-fphydk-rebuildinstance- namespace: ns-cbpst spec: type: RebuildInstance clusterName: mysql-fphydk force: true rebuildFrom: - componentName: mysql instances: - name: mysql-fphydk-mysql-0 backupName: backup-ns-cbpst-mysql-fphydk-20250528130909 inPlace: false restoreEnv: - name: BACKUP_FOR_STANDBY value: "true" check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mysql-fphydk.yaml` opsrequest.operations.kubeblocks.io/mysql-fphydk-rebuildinstance-78dl7 created create test_ops_cluster_mysql-fphydk.yaml Success `rm -rf test_ops_cluster_mysql-fphydk.yaml` check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Running 0/1 May 28,2025 13:11 UTC+0800 ops_status:mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Running 0/1 May 28,2025 13:11 UTC+0800 ops_status:mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Running 0/1 May 28,2025 13:11 UTC+0800 ops_status:mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Running 0/1 May 28,2025 13:11 UTC+0800 ops_status:mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Running 0/1 May 28,2025 13:11 UTC+0800 ops_status:mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Running 0/1 May 28,2025 13:11 UTC+0800 check ops status done ops_status:mysql-fphydk-rebuildinstance-78dl7 ns-cbpst RebuildInstance mysql-fphydk mysql Succeed 1/1 May 28,2025 13:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-rebuildinstance-78dl7 --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-rebuildinstance-78dl7 patched `kbcli cluster delete-ops --name mysql-fphydk-rebuildinstance-78dl7 --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-rebuildinstance-78dl7 deleted check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-2 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 13:11 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-2 --namespace ns-cbpst -- bash ` connect checking... connect checking... check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-2 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-cbpst-mysql-fphydk-20250528130909 --namespace ns-cbpst ` backup.dataprotection.kubeblocks.io/backup-ns-cbpst-mysql-fphydk-20250528130909 patched `kbcli cluster delete-backup mysql-fphydk --name backup-ns-cbpst-mysql-fphydk-20250528130909 --force --auto-approve --namespace ns-cbpst ` Backup backup-ns-cbpst-mysql-fphydk-20250528130909 deleted cluster xtrabackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mysql-fphydk --method xtrabackup --namespace ns-cbpst ` Backup backup-ns-cbpst-mysql-fphydk-20250528131329 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-cbpst-mysql-fphydk-20250528131329 -n ns-cbpst check backup status `kbcli cluster list-backups mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-cbpst-mysql-fphydk-20250528131329 ns-cbpst mysql-fphydk xtrabackup Running Delete May 28,2025 13:13 UTC+0800 backup_status:mysql-fphydk-xtrabackup-Running backup_status:mysql-fphydk-xtrabackup-Running check backup status done backup_status:backup-ns-cbpst-mysql-fphydk-20250528131329 ns-cbpst mysql-fphydk xtrabackup Completed 6674096 10s Delete May 28,2025 13:13 UTC+0800 May 28,2025 13:13 UTC+0800 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql `CREATE DATABASE IF NOT EXISTS kbclidb;CREATE TABLE IF NOT EXISTS kbclidb.msg (id INT UNSIGNED NOT NULL AUTO_INCREMENT,msg TEXT,time TIMESTAMP NULL,PRIMARY KEY (id)) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;insert into kbclidb.msg (msg, time) values ('kbcli-test-data-fphydk0', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql `insert into kbclidb.msg (msg, time) values ('kbcli-test-data-fphydk1', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql `insert into kbclidb.msg (msg, time) values ('kbcli-test-data-fphydk2', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql `insert into kbclidb.msg (msg, time) values ('kbcli-test-data-fphydk3', CURRENT_TIMESTAMP);` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. cluster xtrabackup-inc backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mysql-fphydk --method xtrabackup-inc --namespace ns-cbpst ` Backup backup-ns-cbpst-mysql-fphydk-20250528131453 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-cbpst-mysql-fphydk-20250528131453 -n ns-cbpst check backup status `kbcli cluster list-backups mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-cbpst-mysql-fphydk-20250528131329 ns-cbpst mysql-fphydk xtrabackup Completed 6674096 10s Delete May 28,2025 13:13 UTC+0800 May 28,2025 13:13 UTC+0800 backup-ns-cbpst-mysql-fphydk-20250528131453 ns-cbpst mysql-fphydk xtrabackup-inc Running Delete May 28,2025 13:14 UTC+0800 backup_status:mysql-fphydk-xtrabackup-inc-Running backup_status:mysql-fphydk-xtrabackup-inc-Running check backup status done backup_status:backup-ns-cbpst-mysql-fphydk-20250528131453 ns-cbpst mysql-fphydk xtrabackup-inc Completed 405580 11s Delete May 28,2025 13:14 UTC+0800 May 28,2025 13:15 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "abk-fphydk" not found `kbcli cluster describe-backup --names backup-ns-cbpst-mysql-fphydk-20250528131453 --namespace ns-cbpst ` Name: backup-ns-cbpst-mysql-fphydk-20250528131453 Cluster: mysql-fphydk Namespace: ns-cbpst Spec: Method: xtrabackup-inc Policy Name: mysql-fphydk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-cbpst-mysql-fphydk-20250528131453-ded7e22 TargetPodName: mysql-fphydk-mysql-2 Phase: Completed Start Time: May 28,2025 13:14 UTC+0800 Completion Time: May 28,2025 13:15 UTC+0800 Status: Phase: Completed Total Size: 405580 ActionSet Name: mysql-xtrabackup-inc-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: May 28,2025 13:14 UTC+0800 Completion Time: May 28,2025 13:15 UTC+0800 Path: /ns-cbpst/mysql-fphydk-06b06dde-504d-44aa-b50f-1f5817ef30b3/mysql/backup-ns-cbpst-mysql-fphydk-20250528131453 Warning Events: `kbcli cluster restore abk-fphydk --backup backup-ns-cbpst-mysql-fphydk-20250528131453 --namespace ns-cbpst ` Cluster abk-fphydk created check cluster status `kbcli cluster list abk-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS abk-fphydk ns-cbpst mysql WipeOut Creating May 28,2025 13:15 UTC+0800 clusterdefinition.kubeblocks.io/name=mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances abk-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME abk-fphydk-mysql-0 ns-cbpst abk-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-9-182.us-west-2.compute.internal/172.31.9.182 May 28,2025 13:15 UTC+0800 abk-fphydk-mysql-1 ns-cbpst abk-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 13:16 UTC+0800 check pod status done check cluster role check cluster role done primary: abk-fphydk-mysql-0;secondary: abk-fphydk-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it abk-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check cluster connect done `kbcli cluster describe-backup --names backup-ns-cbpst-mysql-fphydk-20250528131453 --namespace ns-cbpst ` Name: backup-ns-cbpst-mysql-fphydk-20250528131453 Cluster: mysql-fphydk Namespace: ns-cbpst Spec: Method: xtrabackup-inc Policy Name: mysql-fphydk-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-cbpst-mysql-fphydk-20250528131453-ded7e22 TargetPodName: mysql-fphydk-mysql-2 Phase: Completed Start Time: May 28,2025 13:14 UTC+0800 Completion Time: May 28,2025 13:15 UTC+0800 Status: Phase: Completed Total Size: 405580 ActionSet Name: mysql-xtrabackup-inc-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: May 28,2025 13:14 UTC+0800 Completion Time: May 28,2025 13:15 UTC+0800 Path: /ns-cbpst/mysql-fphydk-06b06dde-504d-44aa-b50f-1f5817ef30b3/mysql/backup-ns-cbpst-mysql-fphydk-20250528131453 Warning Events: `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql `select * from kbclidb.msg;` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. id msg time 1 kbcli-test-data-fphydk0 2025-05-28 05:13:52 2 kbcli-test-data-fphydk1 2025-05-28 05:14:00 3 kbcli-test-data-fphydk2 2025-05-28 05:14:16 4 kbcli-test-data-fphydk3 2025-05-28 05:14:32 Point-In-Time Recovery Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql cluster connect `kubectl get secrets -l app.kubernetes.io/instance=abk-fphydk` set secret: abk-fphydk-mysql-account-root `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets abk-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. *************************** 1. row *************************** Slave_IO_State: Waiting for source to send event Master_Host: abk-fphydk-mysql-0.abk-fphydk-mysql-headless Master_User: kbreplicator Master_Port: 3306 Connect_Retry: 60 Master_Log_File: abk-fphydk-mysql-0-bin.000001 Read_Master_Log_Pos: 47876 Relay_Log_File: relay-bin.000002 Relay_Log_Pos: 48118 Relay_Master_Log_File: abk-fphydk-mysql-0-bin.000001 Slave_IO_Running: Yes Slave_SQL_Running: Yes Replicate_Do_DB: Replicate_Ignore_DB: Replicate_Do_Table: Replicate_Ignore_Table: Replicate_Wild_Do_Table: Replicate_Wild_Ignore_Table: Last_Errno: 0 Last_Error: Skip_Counter: 0 Exec_Master_Log_Pos: 47876 Relay_Log_Space: 48322 Until_Condition: None Until_Log_File: Until_Log_Pos: 0 Master_SSL_Allowed: Yes Master_SSL_CA_File: Master_SSL_CA_Path: Master_SSL_Cert: Master_SSL_Cipher: Master_SSL_Key: Seconds_Behind_Master: 0 Master_SSL_Verify_Server_Cert: No Last_IO_Errno: 0 Last_IO_Error: Last_SQL_Errno: 0 Last_SQL_Error: Replicate_Ignore_Server_Ids: Master_Server_Id: 1 Master_UUID: e5ee7977-3b82-11f0-9d07-7ad05bbf7cab Master_Info_File: mysql.slave_master_info SQL_Delay: 0 SQL_Remaining_Delay: NULL Slave_SQL_Running_State: Replica has read all relay log; waiting for more updates Master_Retry_Count: 86400 Master_Bind: Last_IO_Error_Timestamp: Last_SQL_Error_Timestamp: Master_SSL_Crl: Master_SSL_Crlpath: Retrieved_Gtid_Set: e5ee7977-3b82-11f0-9d07-7ad05bbf7cab:1-119 Executed_Gtid_Set: 260e6d87-3b75-11f0-9d5f-d67b54a03f3b:1-2427, 7cc22dd3-3b75-11f0-b535-ce095d4ba1c7:1-3739, e5ee7977-3b82-11f0-9d07-7ad05bbf7cab:1-119 Auto_Position: 1 Replicate_Rewrite_DB: Channel_Name: Master_TLS_Version: Master_public_key_path: Get_master_public_key: 0 Network_Namespace: `echo " echo \"show variables like 'max_connections';\" | mysql -h127.0.0.1 -uroot -pi4Y8Kow879 -P3306 " | kubectl exec -it abk-fphydk-mysql-1 --namespace ns-cbpst -- bash ` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster abk-fphydk `kbcli cluster delete abk-fphydk --auto-approve --namespace ns-cbpst ` Cluster abk-fphydk deleted pod_info:abk-fphydk-mysql-0 4/4 Terminating 0 2m43s abk-fphydk-mysql-1 4/4 Terminating 0 2m20s No resources found in ns-cbpst namespace. delete cluster pod done No resources found in ns-cbpst namespace. check cluster resource non-exist OK: pvc No resources found in ns-cbpst namespace. delete cluster done No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-cbpst-mysql-fphydk-20250528131329 --namespace ns-cbpst ` backup.dataprotection.kubeblocks.io/backup-ns-cbpst-mysql-fphydk-20250528131329 patched `kbcli cluster delete-backup mysql-fphydk --name backup-ns-cbpst-mysql-fphydk-20250528131329 --force --auto-approve --namespace ns-cbpst ` Backup backup-ns-cbpst-mysql-fphydk-20250528131329 deleted `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-cbpst-mysql-fphydk-20250528131453 --namespace ns-cbpst ` backup.dataprotection.kubeblocks.io/backup-ns-cbpst-mysql-fphydk-20250528131453 patched `kbcli cluster delete-backup mysql-fphydk --name backup-ns-cbpst-mysql-fphydk-20250528131453 --force --auto-approve --namespace ns-cbpst ` Backup backup-ns-cbpst-mysql-fphydk-20250528131453 deleted cluster list-logs `kbcli cluster list-logs mysql-fphydk --namespace ns-cbpst ` No log files found. Error from server (NotFound): pods "mysql-fphydk-mysql-1" not found cluster logs `kbcli cluster logs mysql-fphydk --tail 30 --namespace ns-cbpst ` Defaulted container "mysql" out of: mysql, mysql-exporter, kbagent, config-manager, init-data (init), init-xtrabackup (init), init-jemalloc (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) 2025-05-28T05:12:17.906552-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing FLUSH NO_WRITE_TO_BINLOG ENGINE LOGS... 2025-05-28T05:12:17.918866-00:00 0 [Note] [MY-011825] [Xtrabackup] The latest check point (for incremental): '285944818' 2025-05-28T05:12:17.918926-00:00 0 [Note] [MY-011825] [Xtrabackup] Stopping log copying thread at LSN 285947949 2025-05-28T05:12:17.919586-00:00 1 [Note] [MY-011825] [Xtrabackup] >> log scanned up to (285950144) 2025-05-28T05:12:18.922845-00:00 0 [Note] [MY-011825] [Xtrabackup] Executing UNLOCK INSTANCE 2025-05-28T05:12:18.923199-00:00 0 [Note] [MY-011825] [Xtrabackup] All tables unlocked 2025-05-28T05:12:18.923983-00:00 0 [Note] [MY-011825] [Xtrabackup] Compressing and streaming ib_buffer_pool to 2025-05-28T05:12:18.924545-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Compressing and streaming ib_buffer_pool to 2025-05-28T05:12:18.924727-00:00 0 [Note] [MY-011825] [Xtrabackup] Backup created in directory '/xtrabackup_backupfiles/' 2025-05-28T05:12:18.924764-00:00 0 [Note] [MY-011825] [Xtrabackup] MySQL binlog position: filename 'mysql-fphydk-mysql-1-bin.000025', position '638', GTID of the last change '260e6d87-3b75-11f0-9d5f-d67b54a03f3b:1-2427,7cc22dd3-3b75-11f0-b535-ce095d4ba1c7:1-3573' 2025-05-28T05:12:18.925421-00:00 0 [Note] [MY-011825] [Xtrabackup] Compressing and streaming 2025-05-28T05:12:18.925615-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Compressing and streaming file 2025-05-28T05:12:18.926670-00:00 0 [Note] [MY-011825] [Xtrabackup] Compressing and streaming 2025-05-28T05:12:18.926738-00:00 0 [Note] [MY-011825] [Xtrabackup] Done: Compressing and streaming file 2025-05-28T05:12:19.927098-00:00 0 [Note] [MY-011825] [Xtrabackup] Transaction log of lsn (285445816) to (285951310) was copied. 2025-05-28T05:12:20.153986-00:00 0 [Note] [MY-011825] [Xtrabackup] completed OK! 2025-05-28T05:12:25Z INFO HA This member is Cluster's leader 2025-05-28T05:12:25Z DEBUG HA Refresh leader ttl 2025-05-28T05:13:25Z INFO HA This member is Cluster's leader 2025-05-28T05:13:25Z DEBUG HA Refresh leader ttl 2025-05-28T05:14:26Z INFO HA This member is Cluster's leader 2025-05-28T05:14:26Z DEBUG HA Refresh leader ttl 2025-05-28T05:15:26Z INFO HA This member is Cluster's leader 2025-05-28T05:15:26Z DEBUG HA Refresh leader ttl 2025-05-28T05:16:26Z INFO HA This member is Cluster's leader 2025-05-28T05:16:26Z DEBUG HA Refresh leader ttl 2025-05-28T05:17:26Z INFO HA This member is Cluster's leader 2025-05-28T05:17:26Z DEBUG HA Refresh leader ttl 2025-05-28T05:18:26Z INFO HA This member is Cluster's leader 2025-05-28T05:18:26Z DEBUG HA Refresh leader ttl cluster logs slow `kbcli cluster logs mysql-fphydk --tail 30 --file-type=slow --namespace ns-cbpst ` error: can't get file path pattern by type slow cluster logs error `kbcli cluster logs mysql-fphydk --tail 30 --file-type=error --namespace ns-cbpst ` error: can't get file path pattern by type error LB_TYPE is set to: internet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose mysql-fphydk --auto-approve --force=true --type internet --enable false --components mysql --role-selector primary --namespace ns-cbpst ` OpsRequest mysql-fphydk-expose-jbfvd created successfully, you can view the progress: kbcli cluster describe-ops mysql-fphydk-expose-jbfvd -n ns-cbpst check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-expose-jbfvd ns-cbpst Expose mysql-fphydk mysql Running 0/1 May 28,2025 13:19 UTC+0800 check cluster status `kbcli cluster list mysql-fphydk --show-labels --namespace ns-cbpst ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mysql-fphydk ns-cbpst mysql WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=mysql-fphydk,clusterdefinition.kubeblocks.io/name=mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mysql-fphydk --namespace ns-cbpst ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mysql-fphydk-mysql-1 ns-cbpst mysql-fphydk mysql Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-13-93.us-west-2.compute.internal/172.31.13.93 May 28,2025 12:23 UTC+0800 mysql-fphydk-mysql-2 ns-cbpst mysql-fphydk mysql Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:14Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 13:11 UTC+0800 check pod status done check cluster role check cluster role done primary: mysql-fphydk-mysql-1;secondary: mysql-fphydk-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"SHOW SLAVE STATUS\G;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-2 --namespace ns-cbpst -- bash ` check cluster connect done check ops status `kbcli cluster list-ops mysql-fphydk --status all --namespace ns-cbpst ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mysql-fphydk-switchover-nwbml ns-cbpst Switchover mysql-fphydk mysql Succeed 1/1 May 28,2025 12:15 UTC+0800 mysql-fphydk-expose-jbfvd ns-cbpst Expose mysql-fphydk mysql Succeed 1/1 May 28,2025 13:19 UTC+0800 check ops status done ops_status:mysql-fphydk-expose-jbfvd ns-cbpst Expose mysql-fphydk mysql Succeed 1/1 May 28,2025 13:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mysql-fphydk-expose-jbfvd --namespace ns-cbpst ` opsrequest.operations.kubeblocks.io/mysql-fphydk-expose-jbfvd patched `kbcli cluster delete-ops --name mysql-fphydk-expose-jbfvd --force --auto-approve --namespace ns-cbpst ` OpsRequest mysql-fphydk-expose-jbfvd deleted `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hmysql-fphydk-mysql.ns-cbpst.svc.cluster.local -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-1 --namespace ns-cbpst -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mysql-fphydk` set secret: mysql-fphydk-mysql-account-root `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mysql-fphydk-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:i4Y8Kow879;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'i4Y8Kow879' " | kubectl exec -it mysql-fphydk-mysql-2 --namespace ns-cbpst -- bash ` check readonly db_client batch data Success delete cluster mysql-fphydk `kbcli cluster delete mysql-fphydk --auto-approve --namespace ns-cbpst ` Cluster mysql-fphydk deleted pod_info:mysql-fphydk-mysql-1 4/4 Terminating 51 (21m ago) 57m mysql-fphydk-mysql-2 4/4 Terminating 0 8m37s No resources found in ns-cbpst namespace. delete cluster pod done No resources found in ns-cbpst namespace. check cluster resource non-exist OK: pvc No resources found in ns-cbpst namespace. delete cluster done No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. No resources found in ns-cbpst namespace. Mysql Test Suite All Done! --------------------------------------Mysql (Topology = semisync Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=mysql-8.0-1.0.0-alpha.0;ComponentVersion=mysql;ServiceVersion=8.0.30;]|[Description=Create a cluster with the specified component definition mysql-8.0-1.0.0-alpha.0 and component version mysql and service version 8.0.30] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Expose]|[Enable=true;TYPE=internet;ComponentName=mysql]|[Description=Expose Enable the internet service with mysql component] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=mysql]|[Description=] [PASSED]|[VolumeExpansion]|[ComponentName=mysql]|[Description=VolumeExpansion the cluster specify component mysql] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=mysql]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Reconfiguring]|[ComponentName=mysql;binlog_expire_logs_seconds=691200]|[Description=Reconfiguring the cluster specify component mysql set binlog_expire_logs_seconds=691200] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=mysql]|[Description=] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=mysql]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[HorizontalScaling Out]|[ComponentName=mysql]|[Description=HorizontalScaling Out the cluster specify component mysql] [WARNING]|[Operation]|[Succeed Or Failed Soon]|[Description=-] [PASSED]|[HorizontalScaling In]|[ComponentName=mysql]|[Description=HorizontalScaling In the cluster specify component mysql] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[VerticalScaling]|[ComponentName=mysql]|[Description=VerticalScaling the cluster specify component mysql] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=mysql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=mysql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=mysql]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[SwitchOver]|[ComponentName=mysql]|[Description=SwitchOver the cluster specify component mysql] [PASSED]|[Bench]|[ComponentName=mysql]|[Description=Bench the cluster service with mysql component] [PASSED]|[Bench]|[HostType=LB;ComponentName=mysql]|[Description=Bench the cluster LB service with mysql component] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=mysql]|[Description=] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mysql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.30;ComponentVersionTo=8.0.31]|[Description=Upgrade the cluster specify component mysql service version from 8.0.30 to 8.0.31] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.31;ComponentVersionTo=8.0.32]|[Description=Upgrade the cluster specify component mysql service version from 8.0.31 to 8.0.32] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.32;ComponentVersionTo=8.0.33]|[Description=Upgrade the cluster specify component mysql service version from 8.0.32 to 8.0.33] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.33;ComponentVersionTo=8.0.34]|[Description=Upgrade the cluster specify component mysql service version from 8.0.33 to 8.0.34] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.34;ComponentVersionTo=8.0.35]|[Description=Upgrade the cluster specify component mysql service version from 8.0.34 to 8.0.35] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.35;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.35 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.35]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.35] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.35;ComponentVersionTo=8.0.36]|[Description=Upgrade the cluster specify component mysql service version from 8.0.35 to 8.0.36] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.36;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.36 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.36]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.36] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.36;ComponentVersionTo=8.0.37]|[Description=Upgrade the cluster specify component mysql service version from 8.0.36 to 8.0.37] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.37;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.37 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.37]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.37] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.37;ComponentVersionTo=8.0.38]|[Description=Upgrade the cluster specify component mysql service version from 8.0.37 to 8.0.38] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.38;ComponentVersionTo=8.0.39]|[Description=Upgrade the cluster specify component mysql service version from 8.0.38 to 8.0.39] [PASSED]|[Upgrade]|[ComponentName=mysql;ComponentVersionFrom=8.0.39;ComponentVersionTo=8.0.38]|[Description=Upgrade the cluster specify component mysql service version from 8.0.39 to 8.0.38] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=mysql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=mysql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=xtrabackup]|[Description=Delete the Schedule xtrabackup restore cluster] [PASSED]|[Backup]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Backup] [PASSED]|[Restore]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=xtrabackup]|[Description=Delete the xtrabackup restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=mysql]|[Description=Rebuild the cluster instance specify component mysql] [PASSED]|[Backup]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Backup] [PASSED]|[Backup]|[BackupMethod=xtrabackup-inc]|[Description=The cluster xtrabackup-inc Backup] [PASSED]|[Restore Increment]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Restore Increment] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=xtrabackup]|[Description=Delete the xtrabackup restore cluster] [PASSED]|[Expose]|[Disable=true;TYPE=internet;ComponentName=mysql]|[Description=Expose Disable the internet service with mysql component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]