source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-sufpk ` `kubectl create namespace ns-sufpk` namespace/ns-sufpk created create namespace ns-sufpk done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.0` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 127M 0 --:--:-- --:--:-- --:--:-- 127M kbcli installed successfully. Kubernetes: v1.32.3-eks-4096722 KubeBlocks: 1.0.0 kbcli: 1.0.0 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.0 done Kubernetes: v1.32.3-eks-4096722 KubeBlocks: 1.0.0 kbcli: 1.0.0 Kubernetes Env: v1.32.3-eks-4096722 check snapshot controller check snapshot controller done eks default-vsc found POD_RESOURCES: No resources found found default storage class: gp3 KubeBlocks version is:1.0.0 skip upgrade KubeBlocks current KubeBlocks version: 1.0.0 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition check component definition set component name:mysql set component version set component version:apecloud-mysql set service versions:8.0.30 set service versions sorted:8.0.30 set apecloud-mysql component definition set apecloud-mysql component definition apecloud-mysql-1.0.0-alpha.0 set replicas first:3,8.0.30 set replicas third:3,8.0.30 set replicas fourth:3,8.0.30 set minimum cmpv service version set minimum cmpv service version replicas:3,8.0.30 REPORT_COUNT:1 CLUSTER_TOPOLOGY:apecloud-mysql topology apecloud-mysql found in cluster definition apecloud-mysql set apecloud-mysql component definition set apecloud-mysql component definition apecloud-mysql-1.0.0-alpha.0 LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-sufpk namespace. termination_policy:WipeOut create 3 replica WipeOut apecloud-mysql cluster check component definition set component definition by component version check cmpd by labels check cmpd by compDefs set component definition: apecloud-mysql-1.0.0-alpha.0 by component version:apecloud-mysql apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: apemysql-yoyqna namespace: ns-sufpk labels: app.kubernetes.io/instance: apemysql-yoyqna spec: clusterDef: apecloud-mysql topology: apecloud-mysql terminationPolicy: WipeOut componentSpecs: - name: mysql serviceVersion: 8.0.30 disableExporter: true replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_apemysql-yoyqna.yaml` cluster.apps.kubeblocks.io/apemysql-yoyqna created apply test_create_apemysql-yoyqna.yaml Success `rm -rf test_create_apemysql-yoyqna.yaml` check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Creating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Failed cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:28 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-104.us-west-2.compute.internal/172.31.2.104 May 27,2025 12:29 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:30 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check pod apemysql-yoyqna-mysql-0 container_name mysql exist password O8f07dhV651s40Rv check pod apemysql-yoyqna-mysql-0 container_name vtablet exist password O8f07dhV651s40Rv check pod apemysql-yoyqna-mysql-0 container_name kbagent exist password O8f07dhV651s40Rv check pod apemysql-yoyqna-mysql-0 container_name config-manager exist password O8f07dhV651s40Rv No container logs contain secret password. describe cluster `kbcli cluster describe apemysql-yoyqna --namespace ns-sufpk ` Name: apemysql-yoyqna Created Time: May 27,2025 12:28 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-sufpk apecloud-mysql apecloud-mysql Running WipeOut Endpoints: COMPONENT INTERNAL EXTERNAL mysql apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql 8.0.30 apemysql-yoyqna-mysql-0 leader Running us-west-2a ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:28 UTC+0800 mysql 8.0.30 apemysql-yoyqna-mysql-1 follower Running us-west-2a ip-172-31-2-104.us-west-2.compute.internal/172.31.2.104 May 27,2025 12:29 UTC+0800 mysql 8.0.30 apemysql-yoyqna-mysql-2 follower Running us-west-2a ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:30 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc Images: COMPONENT COMPONENT-DEFINITION IMAGE mysql apecloud-mysql-1.0.0-alpha.0 docker.io/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20240330.g94d1caf.15 docker.io/apecloud/apecloud-mysql-scale:0.2.7 docker.io/apecloud/kubeblocks-tools:1.0.0 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-sufpk apemysql-yoyqna `kbcli cluster label apemysql-yoyqna app.kubernetes.io/instance- --namespace ns-sufpk ` `kbcli cluster label apemysql-yoyqna app.kubernetes.io/instance=apemysql-yoyqna --namespace ns-sufpk ` `kbcli cluster label apemysql-yoyqna --list --namespace ns-sufpk ` NAME NAMESPACE LABELS apemysql-yoyqna ns-sufpk app.kubernetes.io/instance=apemysql-yoyqna clusterdefinition.kubeblocks.io/name=apecloud-mysql label cluster app.kubernetes.io/instance=apemysql-yoyqna Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=apemysql-yoyqna --namespace ns-sufpk ` `kbcli cluster label apemysql-yoyqna --list --namespace ns-sufpk ` NAME NAMESPACE LABELS apemysql-yoyqna ns-sufpk app.kubernetes.io/instance=apemysql-yoyqna case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=apecloud-mysql label cluster case.name=kbcli.test1 Success `kbcli cluster label apemysql-yoyqna case.name=kbcli.test2 --overwrite --namespace ns-sufpk ` `kbcli cluster label apemysql-yoyqna --list --namespace ns-sufpk ` NAME NAMESPACE LABELS apemysql-yoyqna ns-sufpk app.kubernetes.io/instance=apemysql-yoyqna case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=apecloud-mysql label cluster case.name=kbcli.test2 Success `kbcli cluster label apemysql-yoyqna case.name- --namespace ns-sufpk ` `kbcli cluster label apemysql-yoyqna --list --namespace ns-sufpk ` NAME NAMESPACE LABELS apemysql-yoyqna ns-sufpk app.kubernetes.io/instance=apemysql-yoyqna clusterdefinition.kubeblocks.io/name=apecloud-mysql delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-yoyqna-mysql-0.apemysql-yoyqna-mysql-headless:13306 Leader YES 0 0 2 apemysql-yoyqna-mysql-1.apemysql-yoyqna-mysql-headless:13306 Follower YES 0 0 3 apemysql-yoyqna-mysql-2.apemysql-yoyqna-mysql-headless:13306 Follower YES 0 0 `echo "echo \"show variables like 'max_connections';\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 83 connect cluster Success set max_connections to 83 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-apemysql-yoyqna" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-apemysql-yoyqna --namespace ns-sufpk ` Error from server (NotFound): pods "test-db-client-executionloop-apemysql-yoyqna" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-apemysql-yoyqna" not found `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-apemysql-yoyqna namespace: ns-sufpk spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local" - "--user" - "root" - "--password" - "O8f07dhV651s40Rv" - "--port" - "3306" - "--dbtype" - "mysql" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-apemysql-yoyqna.yaml` pod/test-db-client-executionloop-apemysql-yoyqna created apply test-db-client-executionloop-apemysql-yoyqna.yaml Success `rm -rf test-db-client-executionloop-apemysql-yoyqna.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 52s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 1/1 Running 0 58s check pod test-db-client-executionloop-apemysql-yoyqna status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-yoyqna 0/1 Completed 0 64s check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:28 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-104.us-west-2.compute.internal/172.31.2.104 May 27,2025 12:29 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:30 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done --host apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local --user root --password O8f07dhV651s40Rv --port 3306 --dbtype mysql --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 04:34:29.557 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255), tinyint_col TINYINT, smallint_col SMALLINT, mediumint_col MEDIUMINT, int_col INT, bigint_col BIGINT, float_col FLOAT, double_col DOUBLE, decimal_col DECIMAL(10, 2), date_col DATE, time_col TIME, datetime_col DATETIME, timestamp_col TIMESTAMP, year_col YEAR, char_col CHAR(10), text_col TEXT, blob_col BLOB, binary_col BINARY(10), varbinary_col VARBINARY(255), enum_col ENUM('Option1', 'Option2', 'Option3'), set_col SET('Value1', 'Value2', 'Value3') ); Execution loop start:INSERT INTO executions_loop.executions_loop_table (value, tinyint_col, smallint_col, mediumint_col, int_col, bigint_col, float_col, double_col, decimal_col, date_col, time_col, datetime_col, timestamp_col, year_col, char_col, text_col, blob_col, binary_col, varbinary_col, enum_col, set_col) VALUES ('executions_loop_test_1', 21, 26858, 6624095, 439297916, 3499890387694257549, 0.9428848, 0.08211039234699735, 25.8976231143645, '2025-05-27', '04:34:30', '2025-05-27 04:34:30.474', CURRENT_TIMESTAMP, 1977, 'dAtTxCdjK1', 'JcJFz4yb8anEWNKgzgmOEPiEvaogFCrS0JRzzt1BXeKlQ5YRqgoMePJKi6Gb1ncrGZLE7QyGnzhwBzcIenT26vHLfgfNCp1KhKgIgMKpW5TpafEJM9MkuGXceLQ7jRnkfWzlXlmPO32XSyHDwluXM6mTBDhXsLrtQA1yMW6VAM6wYx8ggGivz2oX4saMCI69Uh4QDhskN0ulMAbEAk8RChQGRKgH0pwa7b6TSAzFd7ue7ioGzZyxzdvrThf8CMJ', UNHEX(REPLACE('c40d65d5b79bd4000489', ' ', '')), UNHEX(REPLACE('2ae3bde490c9559032c2', ' ', '')), UNHEX(REPLACE('670b15f16d766a7dfb083b5c4bc56ff71174b5944403821507023f10df013cfd268538d368d3351aae3b5d6694717039b4de59a5623c9b4c165ec26ef90886f7066c7f2251a4d93503525cbceb539cbb36d290c4ea8eaddf8c63082a3bf43bc005b4bea8bff6a0cf115c74b48de66f7deede5903c37699926bc7a25c82b133d578b547b60c08614165d8515919e372c0cc6aa2d4db57d6c56ba087bd5280b8a486e3e8dfd9f9bd07818e7dcf144b738bcf4f0bcda3fafdc7766950704f029e22abd2dece8896a1d5583979796500e9f98fca538191189b792ad466dcc5db74b9368f6f32c12516c1d401d791d9ed03deb4b8c3949df88c825a6b092d22e482', ' ', '')), 'Option2', 'Value2' ); [ 1s ] executions total: 14 successful: 14 failed: 0 disconnect: 0 [ 2s ] executions total: 35 successful: 35 failed: 0 disconnect: 0 [ 3s ] executions total: 57 successful: 57 failed: 0 disconnect: 0 [ 4s ] executions total: 78 successful: 78 failed: 0 disconnect: 0 [ 5s ] executions total: 98 successful: 98 failed: 0 disconnect: 0 [ 6s ] executions total: 123 successful: 123 failed: 0 disconnect: 0 [ 7s ] executions total: 147 successful: 147 failed: 0 disconnect: 0 [ 8s ] executions total: 171 successful: 171 failed: 0 disconnect: 0 [ 9s ] executions total: 193 successful: 193 failed: 0 disconnect: 0 [ 10s ] executions total: 214 successful: 214 failed: 0 disconnect: 0 [ 11s ] executions total: 237 successful: 237 failed: 0 disconnect: 0 [ 12s ] executions total: 259 successful: 259 failed: 0 disconnect: 0 [ 13s ] executions total: 280 successful: 280 failed: 0 disconnect: 0 [ 14s ] executions total: 302 successful: 302 failed: 0 disconnect: 0 [ 15s ] executions total: 324 successful: 324 failed: 0 disconnect: 0 [ 16s ] executions total: 351 successful: 351 failed: 0 disconnect: 0 [ 17s ] executions total: 376 successful: 376 failed: 0 disconnect: 0 [ 18s ] executions total: 404 successful: 404 failed: 0 disconnect: 0 [ 19s ] executions total: 427 successful: 427 failed: 0 disconnect: 0 [ 20s ] executions total: 450 successful: 450 failed: 0 disconnect: 0 [ 21s ] executions total: 474 successful: 474 failed: 0 disconnect: 0 [ 22s ] executions total: 489 successful: 489 failed: 0 disconnect: 0 [ 23s ] executions total: 512 successful: 512 failed: 0 disconnect: 0 [ 24s ] executions total: 534 successful: 534 failed: 0 disconnect: 0 [ 25s ] executions total: 552 successful: 552 failed: 0 disconnect: 0 [ 26s ] executions total: 573 successful: 573 failed: 0 disconnect: 0 [ 27s ] executions total: 597 successful: 597 failed: 0 disconnect: 0 [ 28s ] executions total: 626 successful: 626 failed: 0 disconnect: 0 [ 29s ] executions total: 650 successful: 650 failed: 0 disconnect: 0 [ 30s ] executions total: 674 successful: 674 failed: 0 disconnect: 0 [ 31s ] executions total: 693 successful: 693 failed: 0 disconnect: 0 [ 32s ] executions total: 715 successful: 715 failed: 0 disconnect: 0 [ 33s ] executions total: 742 successful: 742 failed: 0 disconnect: 0 [ 34s ] executions total: 766 successful: 766 failed: 0 disconnect: 0 [ 35s ] executions total: 790 successful: 790 failed: 0 disconnect: 0 [ 36s ] executions total: 816 successful: 816 failed: 0 disconnect: 0 [ 37s ] executions total: 842 successful: 842 failed: 0 disconnect: 0 [ 38s ] executions total: 866 successful: 866 failed: 0 disconnect: 0 [ 39s ] executions total: 892 successful: 892 failed: 0 disconnect: 0 [ 40s ] executions total: 918 successful: 918 failed: 0 disconnect: 0 [ 41s ] executions total: 944 successful: 944 failed: 0 disconnect: 0 [ 42s ] executions total: 969 successful: 969 failed: 0 disconnect: 0 [ 43s ] executions total: 996 successful: 996 failed: 0 disconnect: 0 [ 44s ] executions total: 1023 successful: 1023 failed: 0 disconnect: 0 [ 45s ] executions total: 1047 successful: 1047 failed: 0 disconnect: 0 [ 46s ] executions total: 1074 successful: 1074 failed: 0 disconnect: 0 [ 47s ] executions total: 1100 successful: 1100 failed: 0 disconnect: 0 [ 48s ] executions total: 1125 successful: 1125 failed: 0 disconnect: 0 [ 49s ] executions total: 1147 successful: 1147 failed: 0 disconnect: 0 [ 50s ] executions total: 1172 successful: 1172 failed: 0 disconnect: 0 [ 51s ] executions total: 1198 successful: 1198 failed: 0 disconnect: 0 [ 52s ] executions total: 1222 successful: 1222 failed: 0 disconnect: 0 [ 53s ] executions total: 1248 successful: 1248 failed: 0 disconnect: 0 [ 54s ] executions total: 1273 successful: 1273 failed: 0 disconnect: 0 [ 55s ] executions total: 1299 successful: 1299 failed: 0 disconnect: 0 [ 56s ] executions total: 1325 successful: 1325 failed: 0 disconnect: 0 [ 57s ] executions total: 1350 successful: 1350 failed: 0 disconnect: 0 [ 58s ] executions total: 1374 successful: 1374 failed: 0 disconnect: 0 [ 60s ] executions total: 1393 successful: 1393 failed: 0 disconnect: 0 Test Result: Total Executions: 1393 Successful Executions: 1393 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: mysql Host: apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local Port: 3306 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 1393 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-apemysql-yoyqna --namespace ns-sufpk ` pod/test-db-client-executionloop-apemysql-yoyqna patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-apemysql-yoyqna" force deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"CREATE DATABASE IF NOT EXISTS mydb; use mydb; DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); INSERT INTO tmp_table (value) VALUES ('jfkle');\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. add consistent data jfkle Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"CREATE DATABASE IF NOT EXISTS mydb; use mydb; DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); INSERT INTO tmp_table (value) VALUES ('jfkle');\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. ERROR 7504 (HY000) at line 1: The consensus follower is not allowed to to do current operation. command terminated with exit code 1 check add consistent data readonly Success LB_TYPE is set to: intranet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose apemysql-yoyqna --auto-approve --force=true --type intranet --enable true --components mysql --role-selector leader --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-expose-fqtgl created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-expose-fqtgl -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-expose-fqtgl ns-sufpk Expose apemysql-yoyqna mysql Running 0/1 May 27,2025 12:36 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:28 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-104.us-west-2.compute.internal/172.31.2.104 May 27,2025 12:29 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:30 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-expose-fqtgl ns-sufpk Expose apemysql-yoyqna mysql Succeed 1/1 May 27,2025 12:36 UTC+0800 check ops status done ops_status:apemysql-yoyqna-expose-fqtgl ns-sufpk Expose apemysql-yoyqna mysql Succeed 1/1 May 27,2025 12:36 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-expose-fqtgl --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-expose-fqtgl patched `kbcli cluster delete-ops --name apemysql-yoyqna-expose-fqtgl --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-expose-fqtgl deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:28 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-104.us-west-2.compute.internal/172.31.2.104 May 27,2025 12:29 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:30 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1;follower: apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done check failover pod name failover pod name:apemysql-yoyqna-mysql-1 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart apemysql-yoyqna --auto-approve --force=true --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-restart-vfbmc created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-restart-vfbmc -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-restart-vfbmc ns-sufpk Restart apemysql-yoyqna mysql Running 0/3 May 27,2025 12:38 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-restart-vfbmc ns-sufpk Restart apemysql-yoyqna mysql Succeed 3/3 May 27,2025 12:38 UTC+0800 check ops status done ops_status:apemysql-yoyqna-restart-vfbmc ns-sufpk Restart apemysql-yoyqna mysql Succeed 3/3 May 27,2025 12:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-restart-vfbmc --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-restart-vfbmc patched `kbcli cluster delete-ops --name apemysql-yoyqna-restart-vfbmc --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-restart-vfbmc deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-yoyqna" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-apemysql-yoyqna.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-apemysql-yoyqna created apply test-chaos-mesh-dnserror-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-dnserror-apemysql-yoyqna.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-yoyqna" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-yoyqna" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-apemysql-yoyqna.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-apemysql-yoyqna created apply test-chaos-mesh-oom-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-oom-apemysql-yoyqna.yaml` check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-yoyqna" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-1 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster does not need to check monitor currently check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-1;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-apemysql-yoyqna" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-apemysql-yoyqna --namespace ns-sufpk ` Error from server (NotFound): pods "test-db-client-connectionstress-apemysql-yoyqna" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-apemysql-yoyqna" not found `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-apemysql-yoyqna namespace: ns-sufpk spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local" - "--user" - "root" - "--password" - "O8f07dhV651s40Rv" - "--port" - "3306" - "--database" - "mysql" - "--dbtype" - "mysql" - "--test" - "connectionstress" - "--connections" - "83" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-apemysql-yoyqna.yaml` pod/test-db-client-connectionstress-apemysql-yoyqna created apply test-db-client-connectionstress-apemysql-yoyqna.yaml Success `rm -rf test-db-client-connectionstress-apemysql-yoyqna.yaml` check pod status check pod test-db-client-connectionstress-apemysql-yoyqna status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-apemysql-yoyqna 0/1 Completed 0 6s check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-1;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done --host apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local --user root --password O8f07dhV651s40Rv --port 3306 --database mysql --dbtype mysql --test connectionstress --connections 83 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 04:48:26.330 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered 04:48:28.498 [main] DEBUG com.yashandb.log.LoggerFactory -- Logger type is SLF4J Logger. 04:48:28.500 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306/mysql?useSSL=false&allowPublicKeyRetrieval=true Failed to connect to MySQL database: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" Trying with database MySQL. 04:48:28.505 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306?useSSL=false&allowPublicKeyRetrieval=true java.io.IOException: Failed to connect to MySQL database: at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:63) at com.apecloud.dbtester.tester.MySQLTester.connectionStress(MySQLTester.java:111) at com.apecloud.dbtester.tester.TestExecutor.executeTest(TestExecutor.java:34) at OneClient.executeTest(OneClient.java:105) at OneClient.main(OneClient.java:37) Caused by: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:61) ... 4 more 04:48:28.536 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306/mysql?useSSL=false&allowPublicKeyRetrieval=true Failed to connect to MySQL database: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" Trying with database MySQL. 04:48:28.543 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306?useSSL=false&allowPublicKeyRetrieval=true java.io.IOException: Failed to connect to MySQL database: at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:63) at com.apecloud.dbtester.tester.MySQLTester.connectionStress(MySQLTester.java:111) at com.apecloud.dbtester.tester.TestExecutor.executeTest(TestExecutor.java:34) at OneClient.executeTest(OneClient.java:105) at OneClient.main(OneClient.java:37) Caused by: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:61) ... 4 more 04:48:28.636 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306/mysql?useSSL=false&allowPublicKeyRetrieval=true Failed to connect to MySQL database: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" Trying with database MySQL. 04:48:28.638 [main] DEBUG com.yashandb.conf.ConnectionUrl -- JDBC URL must start with "jdbc:yasdb:" but was: jdbc:mysql://apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local:3306?useSSL=false&allowPublicKeyRetrieval=true java.io.IOException: Failed to connect to MySQL database: at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:63) at com.apecloud.dbtester.tester.MySQLTester.connectionStress(MySQLTester.java:111) at com.apecloud.dbtester.tester.TestExecutor.executeTest(TestExecutor.java:34) at OneClient.executeTest(OneClient.java:105) at OneClient.main(OneClient.java:37) Caused by: java.sql.SQLNonTransientConnectionException: Data source rejected establishment of connection, message from server: "Too many connections" at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:110) at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122) at com.mysql.cj.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:829) at com.mysql.cj.jdbc.ConnectionImpl.(ConnectionImpl.java:449) at com.mysql.cj.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:242) at com.mysql.cj.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:198) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.MySQLTester.connect(MySQLTester.java:61) ... 4 more Test Result: null Connection Information: Database Type: mysql Host: apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local Port: 3306 Database: mysql Table: User: root Org: Access Mode: mysql Test Type: connectionstress Connection Count: 83 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-apemysql-yoyqna --namespace ns-sufpk ` pod/test-db-client-connectionstress-apemysql-yoyqna patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-apemysql-yoyqna" force deleted check failover pod name failover pod name:apemysql-yoyqna-mysql-1 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-yoyqna" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-apemysql-yoyqna.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-apemysql-yoyqna created apply test-chaos-mesh-networkbandwidthover-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-apemysql-yoyqna.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-yoyqna" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-yoyqna" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-apemysql-yoyqna.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-apemysql-yoyqna created apply test-chaos-mesh-networklossover-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-networklossover-apemysql-yoyqna.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Failed May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 12:40 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-yoyqna" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod apemysql-yoyqna-mysql-0 --namespace ns-sufpk -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-14-133.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod apemysql-yoyqna-mysql-1 --namespace ns-sufpk -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-1-144.us-west-2.compute.internal success kubectl get pod apemysql-yoyqna-mysql-2 --namespace ns-sufpk -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-7-54.us-west-2.compute.internal success kubectl drain ip-172-31-14-133.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-14-133.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-wcf8d, kube-system/aws-node-wm7h5, kube-system/ebs-csi-node-dxdzl, kube-system/kube-proxy-6276h evicting pod ns-sufpk/apemysql-yoyqna-mysql-0 evicting pod kube-system/ebs-csi-controller-77c455df54-l2xsd evicting pod ns-byfkn/kafka-ludvaw-kafka-combine-0 pod/kafka-ludvaw-kafka-combine-0 evicted pod/apemysql-yoyqna-mysql-0 evicted pod/ebs-csi-controller-77c455df54-l2xsd evicted node/ip-172-31-14-133.us-west-2.compute.internal drained kubectl uncordon ip-172-31-14-133.us-west-2.compute.internal node/ip-172-31-14-133.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:56 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster configure component_tmp: mysql apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-yoyqna-reconfiguring- namespace: ns-sufpk spec: type: Reconfiguring clusterName: apemysql-yoyqna force: true reconfigures: - componentName: mysql parameters: - key: log_error_verbosity value: '3' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-yoyqna.yaml` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-reconfiguring-rkjm2 created create test_ops_cluster_apemysql-yoyqna.yaml Success `rm -rf test_ops_cluster_apemysql-yoyqna.yaml` check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-reconfiguring-rkjm2 ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Running -/- May 27,2025 12:58 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:56 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-reconfiguring-rkjm2 ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 12:58 UTC+0800 check ops status done ops_status:apemysql-yoyqna-reconfiguring-rkjm2 ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 12:58 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-reconfiguring-rkjm2 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-reconfiguring-rkjm2 patched `kbcli cluster delete-ops --name apemysql-yoyqna-reconfiguring-rkjm2 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-reconfiguring-rkjm2 deleted component_config:mysql check config variables `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'log_error_verbosity'\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash | (grep "log_error_verbosity" || true) | awk '***print mysql***' current value_actual: 3 configure:[log_error_verbosity] result actual:[3] equal expected:[3] `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote apemysql-yoyqna --auto-approve --force=true --instance apemysql-yoyqna-mysql-2 --candidate apemysql-yoyqna-mysql-0 --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-switchover-fqjc2 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-switchover-fqjc2 -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-switchover-fqjc2 ns-sufpk Switchover apemysql-yoyqna apemysql-yoyqna-mysql Running 0/1 May 27,2025 12:59 UTC+0800 ops_status:apemysql-yoyqna-switchover-fqjc2 ns-sufpk Switchover apemysql-yoyqna apemysql-yoyqna-mysql Succeed 1/1 May 27,2025 12:59 UTC+0800 ops Switchover|Custom Succeed or Failed Soon check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:56 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-switchover-fqjc2 ns-sufpk Switchover apemysql-yoyqna apemysql-yoyqna-mysql Succeed 1/1 May 27,2025 12:59 UTC+0800 check ops status done ops_status:apemysql-yoyqna-switchover-fqjc2 ns-sufpk Switchover apemysql-yoyqna apemysql-yoyqna-mysql Succeed 1/1 May 27,2025 12:59 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-switchover-fqjc2 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-switchover-fqjc2 patched `kbcli cluster delete-ops --name apemysql-yoyqna-switchover-fqjc2 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-switchover-fqjc2 deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success switchover pod:apemysql-yoyqna-mysql-0 switchover success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-apemysql-yoyqna namespace: ns-sufpk spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:apemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local,user:root,password:O8f07dhV651s40Rv,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-apemysql-yoyqna.yaml` pod/benchtest-apemysql-yoyqna created apply benchtest-apemysql-yoyqna.yaml Success `rm -rf benchtest-apemysql-yoyqna.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 1/1 Running 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 1/1 Running 0 18s check pod benchtest-apemysql-yoyqna status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 0/1 Completed 0 24s `kubectl logs benchtest-apemysql-yoyqna --tail 30 --namespace ns-sufpk ` [ 7s ] thds: 4 tps: 10.01 qps: 201.13 (r/w/o: 136.09/46.03/19.01) lat (ms,99%): 590.56 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 7.99 qps: 161.89 (r/w/o: 111.93/33.98/15.99) lat (ms,99%): 601.29 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 9.00 qps: 176.09 (r/w/o: 125.06/32.02/19.01) lat (ms,99%): 601.29 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 8.99 qps: 180.79 (r/w/o: 126.85/35.96/17.98) lat (ms,99%): 601.29 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 1386 write: 396 other: 198 total: 1980 transactions: 99 (9.61 per sec.) queries: 1980 (192.17 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.3017s total number of events: 99 Latency (ms): min: 113.27 avg: 412.99 max: 699.90 99th percentile: 694.45 sum: 40886.20 Threads fairness: events (avg/stddev): 24.7500/1.09 execution time (avg/stddev): 10.2216/0.04 `kubectl delete pod benchtest-apemysql-yoyqna --force --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-apemysql-yoyqna" force deleted LB_TYPE is set to: intranet No resources found in ns-sufpk namespace. `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-apemysql-yoyqna namespace: ns-sufpk spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:a0abdd63e2a4a462a81a1f7fb482239f-106e4298f360fb06.elb.us-west-2.amazonaws.com,user:root,password:O8f07dhV651s40Rv,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-apemysql-yoyqna.yaml` pod/benchtest-apemysql-yoyqna created apply benchtest-apemysql-yoyqna.yaml Success `rm -rf benchtest-apemysql-yoyqna.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 1/1 Running 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 1/1 Running 0 18s check pod benchtest-apemysql-yoyqna status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-yoyqna 0/1 Completed 0 24s `kubectl logs benchtest-apemysql-yoyqna --tail 30 --namespace ns-sufpk ` [ 7s ] thds: 4 tps: 9.00 qps: 202.99 (r/w/o: 141.00/44.00/18.00) lat (ms,99%): 493.24 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 11.00 qps: 218.03 (r/w/o: 152.02/44.01/22.00) lat (ms,99%): 493.24 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 11.00 qps: 205.99 (r/w/o: 144.00/40.00/22.00) lat (ms,99%): 493.24 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 11.00 qps: 226.98 (r/w/o: 160.99/44.00/22.00) lat (ms,99%): 493.24 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 1554 write: 444 other: 222 total: 2220 transactions: 111 (10.86 per sec.) queries: 2220 (217.24 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.2176s total number of events: 111 Latency (ms): min: 199.11 avg: 365.96 max: 586.75 99th percentile: 502.20 sum: 40621.79 Threads fairness: events (avg/stddev): 27.7500/1.48 execution time (avg/stddev): 10.1554/0.05 `kubectl delete pod benchtest-apemysql-yoyqna --force --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-apemysql-yoyqna" force deleted cluster configure component_tmp: mysql apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-yoyqna-reconfiguring- namespace: ns-sufpk spec: type: Reconfiguring clusterName: apemysql-yoyqna force: true reconfigures: - componentName: mysql parameters: - key: max_connections value: '1000' - key: general_log value: 'OFF' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-yoyqna.yaml` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-reconfiguring-tdhzd created create test_ops_cluster_apemysql-yoyqna.yaml Success `rm -rf test_ops_cluster_apemysql-yoyqna.yaml` check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-reconfiguring-tdhzd ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 13:04 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-13-86.us-west-2.compute.internal/172.31.13.86 May 27,2025 12:56 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 12:41 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 12:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-reconfiguring-tdhzd ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 13:04 UTC+0800 check ops status done ops_status:apemysql-yoyqna-reconfiguring-tdhzd ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 13:04 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-reconfiguring-tdhzd --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-reconfiguring-tdhzd patched `kbcli cluster delete-ops --name apemysql-yoyqna-reconfiguring-tdhzd --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-reconfiguring-tdhzd deleted component_config:mysql check config variables `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'max_connections'\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash | (grep "max_connections" || true) | awk '***print mysql***' current value_actual: 1000 configure:[max_connections] result actual:[1000] equal expected:[1000] `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'general_log'\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash | (grep "general_log" || true) | awk '***print mysql***' current value_actual: OFF configure:[general_log] result actual:[OFF] equal expected:[OFF] `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster configure component_tmp: mysql apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-yoyqna-reconfiguring- namespace: ns-sufpk spec: type: Reconfiguring clusterName: apemysql-yoyqna force: true reconfigures: - componentName: mysql parameters: - key: max_connections value: '2000' - key: general_log value: 'ON' - key: innodb_sort_buffer_size value: '2097152' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-yoyqna.yaml` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-reconfiguring-64k99 created create test_ops_cluster_apemysql-yoyqna.yaml Success `rm -rf test_ops_cluster_apemysql-yoyqna.yaml` check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-reconfiguring-64k99 ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Running -/- May 27,2025 13:06 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:08 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:07 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:06 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-reconfiguring-64k99 ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 13:06 UTC+0800 check ops status done ops_status:apemysql-yoyqna-reconfiguring-64k99 ns-sufpk Reconfiguring apemysql-yoyqna mysql,mysql Succeed -/- May 27,2025 13:06 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-reconfiguring-64k99 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-reconfiguring-64k99 patched `kbcli cluster delete-ops --name apemysql-yoyqna-reconfiguring-64k99 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-reconfiguring-64k99 deleted component_config:mysql check config variables `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'max_connections'\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash | (grep "max_connections" || true) | awk '***print mysql***' current value_actual: 2000 configure:[max_connections] result actual:[2000] equal expected:[2000] `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'general_log'\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash | (grep "general_log" || true) | awk '***print mysql***' current value_actual: ON configure:[general_log] result actual:[ON] equal expected:[ON] `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql mysql mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. echo "echo \"show variables like 'innodb_sort_buffer_size'\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash | (grep "innodb_sort_buffer_size" || true) | awk '***print mysql***' current value_actual: 2097152 configure:[innodb_sort_buffer_size] result actual:[2097152] equal expected:[2097152] `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-yoyqna" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-apemysql-yoyqna.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-apemysql-yoyqna created apply test-chaos-mesh-timeoffset-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-timeoffset-apemysql-yoyqna.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:08 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:07 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:06 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-yoyqna" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-yoyqna" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-apemysql-yoyqna.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-apemysql-yoyqna created apply test-chaos-mesh-networkcorruptover-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-apemysql-yoyqna.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Failed May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:08 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:07 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:06 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-yoyqna" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-0 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale apemysql-yoyqna --auto-approve --force=true --components mysql --cpu 200m --memory 0.6Gi --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-verticalscaling-vdp26 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-verticalscaling-vdp26 -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-verticalscaling-vdp26 ns-sufpk VerticalScaling apemysql-yoyqna mysql Running 0/3 May 27,2025 13:16 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:19 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:18 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-1;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-verticalscaling-vdp26 ns-sufpk VerticalScaling apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:16 UTC+0800 check ops status done ops_status:apemysql-yoyqna-verticalscaling-vdp26 ns-sufpk VerticalScaling apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-verticalscaling-vdp26 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-verticalscaling-vdp26 patched `kbcli cluster delete-ops --name apemysql-yoyqna-verticalscaling-vdp26 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-verticalscaling-vdp26 deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover networkdelayover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelayover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-yoyqna" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelayover-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelayover-apemysql-yoyqna.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelayover-apemysql-yoyqna created apply test-chaos-mesh-networkdelayover-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-networkdelayover-apemysql-yoyqna.yaml` networkdelayover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Failed May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Failed check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:19 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:18 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelayover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-yoyqna" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover networkdelayover Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success skip cluster Upgrade `kubectl get pvc -l app.kubernetes.io/instance=apemysql-yoyqna,apps.kubeblocks.io/component-name=mysql,apps.kubeblocks.io/vct-name=data --namespace ns-sufpk ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in apemysql-yoyqna namespace. `kbcli cluster volume-expand apemysql-yoyqna --auto-approve --force=true --components mysql --volume-claim-templates data --storage 6Gi --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-volumeexpansion-smzzm created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-volumeexpansion-smzzm -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-volumeexpansion-smzzm ns-sufpk VolumeExpansion apemysql-yoyqna mysql Running 0/3 May 27,2025 13:23 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:19 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:18 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done No resources found in apemysql-yoyqna namespace. check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-volumeexpansion-smzzm ns-sufpk VolumeExpansion apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:23 UTC+0800 check ops status done ops_status:apemysql-yoyqna-volumeexpansion-smzzm ns-sufpk VolumeExpansion apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-volumeexpansion-smzzm --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-volumeexpansion-smzzm patched `kbcli cluster delete-ops --name apemysql-yoyqna-volumeexpansion-smzzm --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-volumeexpansion-smzzm deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-yoyqna" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-apemysql-yoyqna.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-apemysql-yoyqna created apply test-chaos-mesh-dnsrandom-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-dnsrandom-apemysql-yoyqna.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:19 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:18 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-yoyqna" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster hscale offline instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-yoyqna-hscaleoffinstance- labels: app.kubernetes.io/instance: apemysql-yoyqna app.kubernetes.io/managed-by: kubeblocks namespace: ns-sufpk spec: type: HorizontalScaling clusterName: apemysql-yoyqna force: true horizontalScaling: - componentName: mysql scaleIn: onlineInstancesToOffline: - apemysql-yoyqna-mysql-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-yoyqna.yaml` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-hscaleoffinstance-w8mgr created create test_ops_cluster_apemysql-yoyqna.yaml Success `rm -rf test_ops_cluster_apemysql-yoyqna.yaml` check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-hscaleoffinstance-w8mgr ns-sufpk HorizontalScaling apemysql-yoyqna mysql Creating -/- May 27,2025 13:28 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:18 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-hscaleoffinstance-w8mgr ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:28 UTC+0800 check ops status done ops_status:apemysql-yoyqna-hscaleoffinstance-w8mgr ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-hscaleoffinstance-w8mgr --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-hscaleoffinstance-w8mgr patched `kbcli cluster delete-ops --name apemysql-yoyqna-hscaleoffinstance-w8mgr --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-hscaleoffinstance-w8mgr deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster hscale online instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-yoyqna-hscaleoninstance- labels: app.kubernetes.io/instance: apemysql-yoyqna app.kubernetes.io/managed-by: kubeblocks namespace: ns-sufpk spec: type: HorizontalScaling clusterName: apemysql-yoyqna force: true horizontalScaling: - componentName: mysql scaleOut: offlineInstancesToOnline: - apemysql-yoyqna-mysql-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-yoyqna.yaml` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-hscaleoninstance-z88dj created create test_ops_cluster_apemysql-yoyqna.yaml Success `rm -rf test_ops_cluster_apemysql-yoyqna.yaml` check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-hscaleoninstance-z88dj ns-sufpk HorizontalScaling apemysql-yoyqna mysql Running 0/1 May 27,2025 13:30 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:30 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:18 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-hscaleoninstance-z88dj ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:30 UTC+0800 check ops status done ops_status:apemysql-yoyqna-hscaleoninstance-z88dj ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-hscaleoninstance-z88dj --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-hscaleoninstance-z88dj patched `kbcli cluster delete-ops --name apemysql-yoyqna-hscaleoninstance-z88dj --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-hscaleoninstance-z88dj deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop apemysql-yoyqna --auto-approve --force=true --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-stop-8pgk7 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-stop-8pgk7 -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-stop-8pgk7 ns-sufpk Stop apemysql-yoyqna mysql Running 0/3 May 27,2025 13:31 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Stopping May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-stop-8pgk7 ns-sufpk Stop apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:31 UTC+0800 check ops status done ops_status:apemysql-yoyqna-stop-8pgk7 ns-sufpk Stop apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-stop-8pgk7 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-stop-8pgk7 patched `kbcli cluster delete-ops --name apemysql-yoyqna-stop-8pgk7 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-stop-8pgk7 deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start apemysql-yoyqna --force=true --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-start-vmqhm created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-start-vmqhm -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-start-vmqhm ns-sufpk Start apemysql-yoyqna mysql Running 0/3 May 27,2025 13:33 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-start-vmqhm ns-sufpk Start apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:33 UTC+0800 check ops status done ops_status:apemysql-yoyqna-start-vmqhm ns-sufpk Start apemysql-yoyqna mysql Succeed 3/3 May 27,2025 13:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-start-vmqhm --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-start-vmqhm patched `kbcli cluster delete-ops --name apemysql-yoyqna-start-vmqhm --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-start-vmqhm deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-yoyqna" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-apemysql-yoyqna.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-apemysql-yoyqna created apply test-chaos-mesh-networkduplicate-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-networkduplicate-apemysql-yoyqna.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-yoyqna" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-yoyqna" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-0 action: partition mode: all target: mode: all selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-apemysql-yoyqna.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-apemysql-yoyqna created apply test-chaos-mesh-networkpartition-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-networkpartition-apemysql-yoyqna.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-yoyqna" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover fullcpuover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-apemysql-yoyqna.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-apemysql-yoyqna created apply test-chaos-mesh-fullcpuover-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-fullcpuover-apemysql-yoyqna.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-apemysql-yoyqna.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-apemysql-yoyqna created apply test-chaos-mesh-fullcpuover-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-fullcpuover-apemysql-yoyqna.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-1;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-yoyqna" not found failover pod name:apemysql-yoyqna-mysql-1 failover fullcpuover Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover delete pod:apemysql-yoyqna-mysql-1 `kubectl delete pod apemysql-yoyqna-mysql-1 --namespace ns-sufpk ` pod "apemysql-yoyqna-mysql-1" deleted check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:47 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done check failover pod name failover pod name:apemysql-yoyqna-mysql-2 failover Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster mysql scale-out check cluster status before ops check cluster status done cluster_status:Running No resources found in apemysql-yoyqna namespace. `kbcli cluster scale-out apemysql-yoyqna --auto-approve --force=true --components mysql --replicas 1 --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-horizontalscaling-qkn72 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-horizontalscaling-qkn72 -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-horizontalscaling-qkn72 ns-sufpk HorizontalScaling apemysql-yoyqna mysql Running 0/1 May 27,2025 13:48 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:47 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 apemysql-yoyqna-mysql-3 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-22.us-west-2.compute.internal/172.31.9.22 May 27,2025 13:48 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-3 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done No resources found in apemysql-yoyqna namespace. check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-horizontalscaling-qkn72 ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:48 UTC+0800 check ops status done ops_status:apemysql-yoyqna-horizontalscaling-qkn72 ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:48 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-horizontalscaling-qkn72 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-horizontalscaling-qkn72 patched `kbcli cluster delete-ops --name apemysql-yoyqna-horizontalscaling-qkn72 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-horizontalscaling-qkn72 deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster mysql scale-in check cluster status before ops check cluster status done cluster_status:Running No resources found in apemysql-yoyqna namespace. `kbcli cluster scale-in apemysql-yoyqna --auto-approve --force=true --components mysql --replicas 1 --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-horizontalscaling-6p6td created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-horizontalscaling-6p6td -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-horizontalscaling-6p6td ns-sufpk HorizontalScaling apemysql-yoyqna mysql Running 0/1 May 27,2025 13:50 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:47 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-2;follower: apemysql-yoyqna-mysql-0 apemysql-yoyqna-mysql-1 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash` check cluster connect done No resources found in apemysql-yoyqna namespace. check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-horizontalscaling-6p6td ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:50 UTC+0800 check ops status done ops_status:apemysql-yoyqna-horizontalscaling-6p6td ns-sufpk HorizontalScaling apemysql-yoyqna mysql Succeed 1/1 May 27,2025 13:50 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-horizontalscaling-6p6td --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-horizontalscaling-6p6td patched `kbcli cluster delete-ops --name apemysql-yoyqna-horizontalscaling-6p6td --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-horizontalscaling-6p6td deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-yoyqna" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-yoyqna" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-apemysql-yoyqna namespace: ns-sufpk spec: selector: namespaces: - ns-sufpk labelSelectors: apps.kubeblocks.io/pod-name: apemysql-yoyqna-mysql-2 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-apemysql-yoyqna.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-apemysql-yoyqna created apply test-chaos-mesh-podfailure-apemysql-yoyqna.yaml Success `rm -rf test-chaos-mesh-podfailure-apemysql-yoyqna.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Updating May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:47 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-apemysql-yoyqna --namespace ns-sufpk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-yoyqna" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-yoyqna" not found check failover pod name failover pod name:apemysql-yoyqna-mysql-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-1 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update apemysql-yoyqna --termination-policy=WipeOut --namespace ns-sufpk ` cluster.apps.kubeblocks.io/apemysql-yoyqna updated (no change) check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-1 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:47 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-1 apemysql-yoyqna-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done cluster volume-snapshot backup `kbcli cluster backup apemysql-yoyqna --method volume-snapshot --namespace ns-sufpk ` Backup backup-ns-sufpk-apemysql-yoyqna-20250527135534 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-sufpk-apemysql-yoyqna-20250527135534 -n ns-sufpk check backup status `kbcli cluster list-backups apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-sufpk-apemysql-yoyqna-20250527135534 ns-sufpk apemysql-yoyqna volume-snapshot Running Delete May 27,2025 13:55 UTC+0800 backup_status:apemysql-yoyqna-volume-snapshot-Running backup_status:apemysql-yoyqna-volume-snapshot-Running backup_status:apemysql-yoyqna-volume-snapshot-Running backup_status:apemysql-yoyqna-volume-snapshot-Running backup_status:apemysql-yoyqna-volume-snapshot-Running backup_status:apemysql-yoyqna-volume-snapshot-Running check backup status done backup_status:backup-ns-sufpk-apemysql-yoyqna-20250527135534 ns-sufpk apemysql-yoyqna volume-snapshot Completed 6Gi 35s Delete May 27,2025 13:55 UTC+0800 May 27,2025 13:56 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "apemysql-yoyqna-backup" not found `kbcli cluster describe-backup --names backup-ns-sufpk-apemysql-yoyqna-20250527135534 --namespace ns-sufpk ` Name: backup-ns-sufpk-apemysql-yoyqna-20250527135534 Cluster: apemysql-yoyqna Namespace: ns-sufpk Spec: Method: volume-snapshot Policy Name: apemysql-yoyqna-mysql-backup-policy Actions: createVolumeSnapshot-0: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2b1becf] goroutine 1 [running]: github.com/apecloud/kbcli/pkg/cmd/dataprotection.PrintBackupObjDescribe(0xc00133e900, 0xc0008d3088) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:480 +0x4cf github.com/apecloud/kbcli/pkg/cmd/dataprotection.DescribeBackups(0xc00133e900, ***0xc001376450?, 0x18fd69b?, 0xc001404fc8?***) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:458 +0x125 github.com/apecloud/kbcli/pkg/cmd/cluster.describeBackups(0x0?, ***0xc00054df40?, 0x0?, 0x5ff7837400000000?***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:204 +0x66 github.com/apecloud/kbcli/pkg/cmd/cluster.NewDescribeBackupCmd.func1(0xc00131db08?, ***0xc00054df40, 0x0, 0x4***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:195 +0xe5 github.com/spf13/cobra.(*Command).execute(0xc00131db08, ***0xc00054d3c0, 0x4, 0x4***) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xab1 github.com/spf13/cobra.(*Command).ExecuteC(0xc001100c08) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 k8s.io/component-base/cli.run(0xc001100c08) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:146 +0x290 k8s.io/component-base/cli.RunNoErrOutput(...) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:84 main.main() /home/runner/work/kbcli/kbcli/cmd/cli/main.go:31 +0x18 `kbcli cluster restore apemysql-yoyqna-backup --backup backup-ns-sufpk-apemysql-yoyqna-20250527135534 --namespace ns-sufpk ` Cluster apemysql-yoyqna-backup created check cluster status `kbcli cluster list apemysql-yoyqna-backup --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna-backup ns-sufpk apecloud-mysql WipeOut Creating May 27,2025 13:56 UTC+0800 clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna-backup --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-backup-mysql-0 ns-sufpk apemysql-yoyqna-backup mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 13:56 UTC+0800 apemysql-yoyqna-backup-mysql-1 ns-sufpk apemysql-yoyqna-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:56 UTC+0800 apemysql-yoyqna-backup-mysql-2 ns-sufpk apemysql-yoyqna-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:57 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-backup-mysql-0;follower: apemysql-yoyqna-backup-mysql-1 apemysql-yoyqna-backup-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna-backup` set secret: apemysql-yoyqna-backup-mysql-account-root `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-backup-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-backup-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-sufpk-apemysql-yoyqna-20250527135534 --namespace ns-sufpk ` Name: backup-ns-sufpk-apemysql-yoyqna-20250527135534 Cluster: apemysql-yoyqna Namespace: ns-sufpk Spec: Method: volume-snapshot Policy Name: apemysql-yoyqna-mysql-backup-policy Actions: createVolumeSnapshot-0: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2b1becf] goroutine 1 [running]: github.com/apecloud/kbcli/pkg/cmd/dataprotection.PrintBackupObjDescribe(0xc0011d6600, 0xc00090f608) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:480 +0x4cf github.com/apecloud/kbcli/pkg/cmd/dataprotection.DescribeBackups(0xc0011d6600, ***0xc001048d60?, 0x18fd69b?, 0xc0010ec488?***) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:458 +0x125 github.com/apecloud/kbcli/pkg/cmd/cluster.describeBackups(0x0?, ***0xc0008122c0?, 0x0?, 0x57b6c21300000000?***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:204 +0x66 github.com/apecloud/kbcli/pkg/cmd/cluster.NewDescribeBackupCmd.func1(0xc0010b7208?, ***0xc0008122c0, 0x0, 0x4***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:195 +0xe5 github.com/spf13/cobra.(*Command).execute(0xc0010b7208, ***0xc000812280, 0x4, 0x4***) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xab1 github.com/spf13/cobra.(*Command).ExecuteC(0xc000a6f508) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 k8s.io/component-base/cli.run(0xc000a6f508) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:146 +0x290 k8s.io/component-base/cli.RunNoErrOutput(...) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:84 main.main() /home/runner/work/kbcli/kbcli/cmd/cli/main.go:31 +0x18 cluster connect `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna-backup` set secret: apemysql-yoyqna-backup-mysql-account-root `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-yoyqna-backup-mysql-0.apemysql-yoyqna-backup-mysql-headless:13306 Leader YES 0 0 2 apemysql-yoyqna-backup-mysql-1.apemysql-yoyqna-backup-mysql-headless:13306 Follower YES 0 1 3 apemysql-yoyqna-backup-mysql-2.apemysql-yoyqna-backup-mysql-headless:13306 Follower YES 0 1 `echo "echo \"show variables like 'max_connections';\" | mysql -P3306 -hapemysql-yoyqna-backup-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-backup-mysql-0 --namespace ns-sufpk -- bash ` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster apemysql-yoyqna-backup `kbcli cluster delete apemysql-yoyqna-backup --auto-approve --namespace ns-sufpk ` Cluster apemysql-yoyqna-backup deleted pod_info:apemysql-yoyqna-backup-mysql-0 4/4 Terminating 0 94s apemysql-yoyqna-backup-mysql-1 4/4 Terminating 0 81s apemysql-yoyqna-backup-mysql-2 4/4 Terminating 0 67s pod_info:apemysql-yoyqna-backup-mysql-0 4/4 Terminating 0 115s apemysql-yoyqna-backup-mysql-1 4/4 Terminating 0 102s apemysql-yoyqna-backup-mysql-2 4/4 Terminating 0 88s No resources found in ns-sufpk namespace. delete cluster pod done No resources found in ns-sufpk namespace. check cluster resource non-exist OK: pvc No resources found in ns-sufpk namespace. delete cluster done No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-sufpk-apemysql-yoyqna-20250527135534 --namespace ns-sufpk ` backup.dataprotection.kubeblocks.io/backup-ns-sufpk-apemysql-yoyqna-20250527135534 patched `kbcli cluster delete-backup apemysql-yoyqna --name backup-ns-sufpk-apemysql-yoyqna-20250527135534 --force --auto-approve --namespace ns-sufpk ` Backup backup-ns-sufpk-apemysql-yoyqna-20250527135534 deleted No opsrequests found in ns-sufpk namespace. cluster xtrabackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-n8v78 -n kb-odtgx -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-n8v78 -n kb-odtgx -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-odtgx get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-odtgx -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-odtgx -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-odtgx.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-odtgx.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5897fcbdd5-vrzrs --namespace kb-odtgx -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup apemysql-yoyqna --method xtrabackup --namespace ns-sufpk ` Backup backup-ns-sufpk-apemysql-yoyqna-20250527135927 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-sufpk-apemysql-yoyqna-20250527135927 -n ns-sufpk check backup status `kbcli cluster list-backups apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-sufpk-apemysql-yoyqna-20250527135927 ns-sufpk apemysql-yoyqna xtrabackup Running Delete May 27,2025 13:59 UTC+0800 backup_status:apemysql-yoyqna-xtrabackup-Running backup_status:apemysql-yoyqna-xtrabackup-Running backup_status:apemysql-yoyqna-xtrabackup-Running check backup status done backup_status:backup-ns-sufpk-apemysql-yoyqna-20250527135927 ns-sufpk apemysql-yoyqna xtrabackup Completed 8013429 15s Delete May 27,2025 13:59 UTC+0800 May 27,2025 13:59 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "apemysql-yoyqna-backup" not found `kbcli cluster describe-backup --names backup-ns-sufpk-apemysql-yoyqna-20250527135927 --namespace ns-sufpk ` Name: backup-ns-sufpk-apemysql-yoyqna-20250527135927 Cluster: apemysql-yoyqna Namespace: ns-sufpk Spec: Method: xtrabackup Policy Name: apemysql-yoyqna-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-sufpk-apemysql-yoyqna-20250527135927-ace7 TargetPodName: apemysql-yoyqna-mysql-1 Phase: Completed Start Time: May 27,2025 13:59 UTC+0800 Completion Time: May 27,2025 13:59 UTC+0800 Status: Phase: Completed Total Size: 8013429 ActionSet Name: apecloud-mysql-xtrabackup Repository: backuprepo-kbcli-test Duration: 15s Start Time: May 27,2025 13:59 UTC+0800 Completion Time: May 27,2025 13:59 UTC+0800 Path: /ns-sufpk/apemysql-yoyqna-ae4a6b90-cf30-4427-bce2-ce66a76b4b52/mysql/backup-ns-sufpk-apemysql-yoyqna-20250527135927 Time Range Start: May 27,2025 13:59 UTC+0800 Time Range End: May 27,2025 13:59 UTC+0800 Warning Events: `kbcli cluster restore apemysql-yoyqna-backup --backup backup-ns-sufpk-apemysql-yoyqna-20250527135927 --namespace ns-sufpk ` Cluster apemysql-yoyqna-backup created check cluster status `kbcli cluster list apemysql-yoyqna-backup --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna-backup ns-sufpk apecloud-mysql WipeOut Creating May 27,2025 13:59 UTC+0800 clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna-backup --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-backup-mysql-0 ns-sufpk apemysql-yoyqna-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 14:00 UTC+0800 apemysql-yoyqna-backup-mysql-1 ns-sufpk apemysql-yoyqna-backup mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 14:00 UTC+0800 apemysql-yoyqna-backup-mysql-2 ns-sufpk apemysql-yoyqna-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 14:00 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-backup-mysql-1;follower: apemysql-yoyqna-backup-mysql-0 apemysql-yoyqna-backup-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna-backup` set secret: apemysql-yoyqna-backup-mysql-account-root `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-backup-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-backup-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-sufpk-apemysql-yoyqna-20250527135927 --namespace ns-sufpk ` Name: backup-ns-sufpk-apemysql-yoyqna-20250527135927 Cluster: apemysql-yoyqna Namespace: ns-sufpk Spec: Method: xtrabackup Policy Name: apemysql-yoyqna-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-sufpk-apemysql-yoyqna-20250527135927-ace7 TargetPodName: apemysql-yoyqna-mysql-1 Phase: Completed Start Time: May 27,2025 13:59 UTC+0800 Completion Time: May 27,2025 13:59 UTC+0800 Status: Phase: Completed Total Size: 8013429 ActionSet Name: apecloud-mysql-xtrabackup Repository: backuprepo-kbcli-test Duration: 15s Start Time: May 27,2025 13:59 UTC+0800 Completion Time: May 27,2025 13:59 UTC+0800 Path: /ns-sufpk/apemysql-yoyqna-ae4a6b90-cf30-4427-bce2-ce66a76b4b52/mysql/backup-ns-sufpk-apemysql-yoyqna-20250527135927 Time Range Start: May 27,2025 13:59 UTC+0800 Time Range End: May 27,2025 13:59 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna-backup` set secret: apemysql-yoyqna-backup-mysql-account-root `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-yoyqna-backup-mysql-0.apemysql-yoyqna-backup-mysql-headless:13306 Follower YES 0 0 2 apemysql-yoyqna-backup-mysql-1.apemysql-yoyqna-backup-mysql-headless:13306 Leader YES 0 0 3 apemysql-yoyqna-backup-mysql-2.apemysql-yoyqna-backup-mysql-headless:13306 Follower YES 0 1 `echo "echo \"show variables like 'max_connections';\" | mysql -P3306 -hapemysql-yoyqna-backup-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-backup-mysql-1 --namespace ns-sufpk -- bash ` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster apemysql-yoyqna-backup `kbcli cluster delete apemysql-yoyqna-backup --auto-approve --namespace ns-sufpk ` Cluster apemysql-yoyqna-backup deleted pod_info:apemysql-yoyqna-backup-mysql-0 4/4 Terminating 0 106s apemysql-yoyqna-backup-mysql-1 4/4 Terminating 0 84s apemysql-yoyqna-backup-mysql-2 4/4 Terminating 0 70s pod_info:apemysql-yoyqna-backup-mysql-0 4/4 Terminating 0 2m7s apemysql-yoyqna-backup-mysql-1 4/4 Terminating 0 105s apemysql-yoyqna-backup-mysql-2 4/4 Terminating 0 91s No resources found in ns-sufpk namespace. delete cluster pod done No resources found in ns-sufpk namespace. check cluster resource non-exist OK: pvc No resources found in ns-sufpk namespace. delete cluster done No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-yoyqna-rebuildinstance- namespace: ns-sufpk spec: type: RebuildInstance clusterName: apemysql-yoyqna force: true rebuildFrom: - componentName: mysql instances: - name: apemysql-yoyqna-mysql-1 backupName: backup-ns-sufpk-apemysql-yoyqna-20250527135927 inPlace: false restoreEnv: - name: SIGNAL_FILE value: .xtrabackup_rebuild check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-yoyqna.yaml` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-rebuildinstance-sdbc8 created create test_ops_cluster_apemysql-yoyqna.yaml Success `rm -rf test_ops_cluster_apemysql-yoyqna.yaml` check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Running 0/1 May 27,2025 14:02 UTC+0800 check ops status done ops_status:apemysql-yoyqna-rebuildinstance-sdbc8 ns-sufpk RebuildInstance apemysql-yoyqna mysql Succeed 1/1 May 27,2025 14:02 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-rebuildinstance-sdbc8 --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-rebuildinstance-sdbc8 patched `kbcli cluster delete-ops --name apemysql-yoyqna-rebuildinstance-sdbc8 --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-rebuildinstance-sdbc8 deleted check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 apemysql-yoyqna-mysql-3 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-38.us-west-2.compute.internal/172.31.12.38 May 27,2025 14:03 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-2 apemysql-yoyqna-mysql-3 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-sufpk-apemysql-yoyqna-20250527135927 --namespace ns-sufpk ` backup.dataprotection.kubeblocks.io/backup-ns-sufpk-apemysql-yoyqna-20250527135927 patched `kbcli cluster delete-backup apemysql-yoyqna --name backup-ns-sufpk-apemysql-yoyqna-20250527135927 --force --auto-approve --namespace ns-sufpk ` Backup backup-ns-sufpk-apemysql-yoyqna-20250527135927 deleted No opsrequests found in ns-sufpk namespace. `kubectl get backupschedule -l app.kubernetes.io/instance=apemysql-yoyqna ` `kubectl get backupschedule apemysql-yoyqna-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched check backup status `kbcli cluster list-backups apemysql-yoyqna --namespace ns-sufpk ` No backups found in ns-sufpk namespace. No backups found in ns-sufpk namespace. backup_status:-- No backups found in ns-sufpk namespace. backup_status:-- No backups found in ns-sufpk namespace. backup_status:-- No backups found in ns-sufpk namespace. backup_status:-- No backups found in ns-sufpk namespace. backup_status:-- No backups found in ns-sufpk namespace. backup_status:-- No backups found in ns-sufpk namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=apemysql-yoyqna ` `kubectl get backupschedule apemysql-yoyqna-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backup_status:apemysql-yoyqna-xtrabackup-Running backup_status:apemysql-yoyqna-xtrabackup-Running backup_status:apemysql-yoyqna-xtrabackup-Running check backup status done backup_status:apemysql-yoyqna-xtrabackup-20250527060600 ns-sufpk apemysql-yoyqna xtrabackup Completed 5976052 25s Delete May 27,2025 14:06 UTC+0800 May 27,2025 14:06 UTC+0800 Jun 03,2025 14:06 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=apemysql-yoyqna ` `kubectl get backupschedule apemysql-yoyqna-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-yoyqna-mysql-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "apemysql-yoyqna-backup" not found `kbcli cluster describe-backup --names apemysql-yoyqna-xtrabackup-20250527060600 --namespace ns-sufpk ` Name: apemysql-yoyqna-xtrabackup-20250527060600 Cluster: apemysql-yoyqna Namespace: ns-sufpk Spec: Method: xtrabackup Policy Name: apemysql-yoyqna-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-apemysql-yoyqna-xtrabackup-20250527060600-c042c714 TargetPodName: apemysql-yoyqna-mysql-2 Phase: Completed Start Time: May 27,2025 14:06 UTC+0800 Completion Time: May 27,2025 14:06 UTC+0800 Status: Phase: Completed Total Size: 5976052 ActionSet Name: apecloud-mysql-xtrabackup Repository: backuprepo-kbcli-test Duration: 25s Expiration Time: Jun 03,2025 14:06 UTC+0800 Start Time: May 27,2025 14:06 UTC+0800 Completion Time: May 27,2025 14:06 UTC+0800 Path: /ns-sufpk/apemysql-yoyqna-ae4a6b90-cf30-4427-bce2-ce66a76b4b52/mysql/apemysql-yoyqna-xtrabackup-20250527060600 Time Range Start: May 27,2025 14:06 UTC+0800 Time Range End: May 27,2025 14:06 UTC+0800 Warning Events: `kbcli cluster restore apemysql-yoyqna-backup --backup apemysql-yoyqna-xtrabackup-20250527060600 --namespace ns-sufpk ` Cluster apemysql-yoyqna-backup created check cluster status `kbcli cluster list apemysql-yoyqna-backup --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna-backup ns-sufpk apecloud-mysql WipeOut Creating May 27,2025 14:06 UTC+0800 clusterdefinition.kubeblocks.io/name=apecloud-mysql cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna-backup --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-backup-mysql-0 ns-sufpk apemysql-yoyqna-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-144.us-west-2.compute.internal/172.31.12.144 May 27,2025 14:07 UTC+0800 apemysql-yoyqna-backup-mysql-1 ns-sufpk apemysql-yoyqna-backup mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-7-54.us-west-2.compute.internal/172.31.7.54 May 27,2025 14:07 UTC+0800 apemysql-yoyqna-backup-mysql-2 ns-sufpk apemysql-yoyqna-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 14:07 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-backup-mysql-1;follower: apemysql-yoyqna-backup-mysql-0 apemysql-yoyqna-backup-mysql-2 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna-backup` set secret: apemysql-yoyqna-backup-mysql-account-root `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-backup-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-backup-mysql-1 --namespace ns-sufpk -- bash` check cluster connect done `kbcli cluster describe-backup --names apemysql-yoyqna-xtrabackup-20250527060600 --namespace ns-sufpk ` Name: apemysql-yoyqna-xtrabackup-20250527060600 Cluster: apemysql-yoyqna Namespace: ns-sufpk Spec: Method: xtrabackup Policy Name: apemysql-yoyqna-mysql-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-apemysql-yoyqna-xtrabackup-20250527060600-c042c714 TargetPodName: apemysql-yoyqna-mysql-2 Phase: Completed Start Time: May 27,2025 14:06 UTC+0800 Completion Time: May 27,2025 14:06 UTC+0800 Status: Phase: Completed Total Size: 5976052 ActionSet Name: apecloud-mysql-xtrabackup Repository: backuprepo-kbcli-test Duration: 25s Expiration Time: Jun 03,2025 14:06 UTC+0800 Start Time: May 27,2025 14:06 UTC+0800 Completion Time: May 27,2025 14:06 UTC+0800 Path: /ns-sufpk/apemysql-yoyqna-ae4a6b90-cf30-4427-bce2-ce66a76b4b52/mysql/apemysql-yoyqna-xtrabackup-20250527060600 Time Range Start: May 27,2025 14:06 UTC+0800 Time Range End: May 27,2025 14:06 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna-backup` set secret: apemysql-yoyqna-backup-mysql-account-root `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-backup-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-yoyqna-backup-mysql-0.apemysql-yoyqna-backup-mysql-headless:13306 Follower YES 0 0 2 apemysql-yoyqna-backup-mysql-1.apemysql-yoyqna-backup-mysql-headless:13306 Leader YES 0 0 3 apemysql-yoyqna-backup-mysql-2.apemysql-yoyqna-backup-mysql-headless:13306 Follower YES 0 1 `echo "echo \"show variables like 'max_connections';\" | mysql -P3306 -hapemysql-yoyqna-backup-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-backup-mysql-1 --namespace ns-sufpk -- bash ` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster apemysql-yoyqna-backup `kbcli cluster delete apemysql-yoyqna-backup --auto-approve --namespace ns-sufpk ` Cluster apemysql-yoyqna-backup deleted pod_info:apemysql-yoyqna-backup-mysql-0 4/4 Terminating 0 113s apemysql-yoyqna-backup-mysql-1 4/4 Terminating 0 81s apemysql-yoyqna-backup-mysql-2 4/4 Terminating 0 65s pod_info:apemysql-yoyqna-backup-mysql-0 4/4 Terminating 0 2m14s apemysql-yoyqna-backup-mysql-1 4/4 Terminating 0 102s apemysql-yoyqna-backup-mysql-2 4/4 Terminating 0 86s No resources found in ns-sufpk namespace. delete cluster pod done No resources found in ns-sufpk namespace. check cluster resource non-exist OK: pvc No resources found in ns-sufpk namespace. delete cluster done No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups apemysql-yoyqna-xtrabackup-20250527060600 --namespace ns-sufpk ` backup.dataprotection.kubeblocks.io/apemysql-yoyqna-xtrabackup-20250527060600 patched `kbcli cluster delete-backup apemysql-yoyqna --name apemysql-yoyqna-xtrabackup-20250527060600 --force --auto-approve --namespace ns-sufpk ` Backup apemysql-yoyqna-xtrabackup-20250527060600 deleted No opsrequests found in ns-sufpk namespace. cluster list-logs `kbcli cluster list-logs apemysql-yoyqna --namespace ns-sufpk ` No log files found. Error from server (NotFound): pods "apemysql-yoyqna-mysql-0" not found cluster logs `kbcli cluster logs apemysql-yoyqna --tail 30 --namespace ns-sufpk ` Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) 2025-05-27T05:58:25Z INFO HA This member is Cluster's leader 2025-05-27T05:58:25Z DEBUG HA Refresh leader ttl 2025-05-27T05:59:25Z INFO HA This member is Cluster's leader 2025-05-27T05:59:25Z DEBUG HA Refresh leader ttl 2025-05-27T06:00:25Z INFO HA This member is Cluster's leader 2025-05-27T06:00:25Z DEBUG HA Refresh leader ttl 2025-05-27T06:01:25Z INFO HA This member is Cluster's leader 2025-05-27T06:01:25Z DEBUG HA Refresh leader ttl 2025-05-27T06:02:25Z INFO HA This member is Cluster's leader 2025-05-27T06:02:25Z DEBUG HA Refresh leader ttl 2025-05-27T06:03:05Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=apemysql-yoyqna,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2025-05-27T06:03:05Z INFO DCS-K8S podlist: 4 2025-05-27T06:03:15Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=apemysql-yoyqna,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2025-05-27T06:03:16Z INFO DCS-K8S podlist: 4 2025-05-27T06:03:26Z INFO HA This member is Cluster's leader 2025-05-27T06:03:26Z DEBUG HA Refresh leader ttl 2025-05-27T06:04:08Z INFO DCS-K8S pod selector: app.kubernetes.io/instance=apemysql-yoyqna,app.kubernetes.io/managed-by=kubeblocks,apps.kubeblocks.io/component-name=mysql 2025-05-27T06:04:08Z INFO DCS-K8S podlist: 3 2025-05-27T06:04:26Z INFO HA This member is Cluster's leader 2025-05-27T06:04:26Z DEBUG HA Refresh leader ttl 2025-05-27T06:05:26Z INFO HA This member is Cluster's leader 2025-05-27T06:05:26Z DEBUG HA Refresh leader ttl 2025-05-27T06:06:26Z INFO HA This member is Cluster's leader 2025-05-27T06:06:26Z DEBUG HA Refresh leader ttl 2025-05-27T06:07:26Z INFO HA This member is Cluster's leader 2025-05-27T06:07:26Z DEBUG HA Refresh leader ttl 2025-05-27T06:08:26Z INFO HA This member is Cluster's leader 2025-05-27T06:08:26Z DEBUG HA Refresh leader ttl 2025-05-27T06:09:26Z INFO HA This member is Cluster's leader 2025-05-27T06:09:26Z DEBUG HA Refresh leader ttl cluster logs slow `kbcli cluster logs apemysql-yoyqna --tail 30 --file-type=slow --namespace ns-sufpk ` /usr/sbin/mysqld, Version: 8.0.30 (WeSQL Server - GPL, Release 5, Revision 94d1caf). started with: Tcp port: 3306 Unix socket: /var/run/mysqld/mysqld.sock Time Id Command Argument cluster logs error `kbcli cluster logs apemysql-yoyqna --tail 30 --file-type=error --namespace ns-sufpk ` 2025-05-27T06:09:40.983900Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 141154 7171 2025-05-27T06:09:41.984098Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 141529 7172 2025-05-27T06:09:42.984285Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 141904 7173 2025-05-27T06:09:43.984458Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 142279 7174 2025-05-27T06:09:44.984646Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 142654 7175 2025-05-27T06:09:45.984814Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 143029 7176 2025-05-27T06:09:46.984988Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 143404 7177 2025-05-27T06:09:47.985158Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 143779 7178 2025-05-27T06:09:48.985330Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 144154 7179 2025-05-27T06:09:49.985497Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 144529 7180 2025-05-27T06:09:50.985683Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 144904 7181 2025-05-27T06:09:51.985885Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 145279 7182 2025-05-27T06:09:52.986091Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 145654 7183 2025-05-27T06:09:53.986263Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 146029 7184 2025-05-27T06:09:54.986463Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 146404 7185 2025-05-27T06:09:55.986714Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 146779 7186 2025-05-27T06:09:56.986952Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 147154 7187 2025-05-27T06:09:57.987162Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 147529 7188 2025-05-27T06:09:58.987490Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 147904 7189 2025-05-27T06:09:59.987678Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 148279 7190 2025-05-27T06:10:00.987894Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 148654 7191 2025-05-27T06:10:01.988103Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 149029 7192 2025-05-27T06:10:02.988302Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 149404 7193 2025-05-27T06:10:03.988532Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 149779 7194 2025-05-27T06:10:04.988738Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 150154 7195 2025-05-27T06:10:05.988908Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 150529 7196 2025-05-27T06:10:06.989112Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 150904 7197 2025-05-27T06:10:07.989314Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 151279 7198 2025-05-27T06:10:08.989526Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 151654 7199 2025-05-27T06:10:09.989869Z 0 [Note] [MY-000000] [Server] Report binlog commit position: /data/mysql/binlog/mysql-bin.000003 152029 7200 LB_TYPE is set to: intranet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose apemysql-yoyqna --auto-approve --force=true --type intranet --enable false --components mysql --role-selector leader --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-expose-vpk6f created successfully, you can view the progress: kbcli cluster describe-ops apemysql-yoyqna-expose-vpk6f -n ns-sufpk check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-expose-vpk6f ns-sufpk Expose apemysql-yoyqna mysql Running 0/1 May 27,2025 14:10 UTC+0800 check cluster status `kbcli cluster list apemysql-yoyqna --show-labels --namespace ns-sufpk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-yoyqna ns-sufpk apecloud-mysql WipeOut Running May 27,2025 12:28 UTC+0800 app.kubernetes.io/instance=apemysql-yoyqna,clusterdefinition.kubeblocks.io/name=apecloud-mysql check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-yoyqna --namespace ns-sufpk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-yoyqna-mysql-0 ns-sufpk apemysql-yoyqna mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-14-133.us-west-2.compute.internal/172.31.14.133 May 27,2025 13:33 UTC+0800 apemysql-yoyqna-mysql-2 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-1-144.us-west-2.compute.internal/172.31.1.144 May 27,2025 13:34 UTC+0800 apemysql-yoyqna-mysql-3 ns-sufpk apemysql-yoyqna mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-38.us-west-2.compute.internal/172.31.12.38 May 27,2025 14:03 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-yoyqna-mysql-0;follower: apemysql-yoyqna-mysql-2 apemysql-yoyqna-mysql-3 `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check cluster connect `echo " echo \"\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash` check cluster connect done check ops status `kbcli cluster list-ops apemysql-yoyqna --status all --namespace ns-sufpk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-yoyqna-expose-vpk6f ns-sufpk Expose apemysql-yoyqna mysql Succeed 1/1 May 27,2025 14:10 UTC+0800 check ops status done ops_status:apemysql-yoyqna-expose-vpk6f ns-sufpk Expose apemysql-yoyqna mysql Succeed 1/1 May 27,2025 14:10 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations apemysql-yoyqna-expose-vpk6f --namespace ns-sufpk ` opsrequest.operations.kubeblocks.io/apemysql-yoyqna-expose-vpk6f patched `kbcli cluster delete-ops --name apemysql-yoyqna-expose-vpk6f --force --auto-approve --namespace ns-sufpk ` OpsRequest apemysql-yoyqna-expose-vpk6f deleted `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv'" | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, vtablet, kbagent, config-manager, init-data (init), init-syncer (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value jfkle check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-yoyqna-mysql.ns-sufpk.svc.cluster.local -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-0 --namespace ns-sufpk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-yoyqna` set secret: apemysql-yoyqna-mysql-account-root `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-yoyqna-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:O8f07dhV651s40Rv;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'O8f07dhV651s40Rv' " | kubectl exec -it apemysql-yoyqna-mysql-2 --namespace ns-sufpk -- bash ` check readonly db_client batch data Success delete cluster apemysql-yoyqna `kbcli cluster delete apemysql-yoyqna --auto-approve --namespace ns-sufpk ` Cluster apemysql-yoyqna deleted pod_info:apemysql-yoyqna-mysql-0 4/4 Terminating 0 37m apemysql-yoyqna-mysql-2 4/4 Terminating 17 (17m ago) 37m apemysql-yoyqna-mysql-3 4/4 Terminating 0 8m20s pod_info:apemysql-yoyqna-mysql-0 4/4 Terminating 0 38m apemysql-yoyqna-mysql-2 4/4 Terminating 17 (17m ago) 37m apemysql-yoyqna-mysql-3 4/4 Terminating 0 8m41s No resources found in ns-sufpk namespace. delete cluster pod done No resources found in ns-sufpk namespace. check cluster resource non-exist OK: pvc No resources found in ns-sufpk namespace. delete cluster done No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. No resources found in ns-sufpk namespace. ApeCloud Mysql Test Suite All Done! --------------------------------------ApeCloud Mysql (Topology = apecloud-mysql Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=apecloud-mysql-1.0.0-alpha.0;ComponentVersion=apecloud-mysql;ServiceVersion=8.0.30;]|[Description=Create a cluster with the specified component definition apecloud-mysql-1.0.0-alpha.0 and component version apecloud-mysql and service version 8.0.30] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=jfkle]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=jfkle;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Expose]|[Enable=true;TYPE=intranet;ComponentName=mysql]|[Description=Expose Enable the intranet service with mysql component] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=mysql]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=mysql]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=mysql]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=mysql]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=mysql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Reconfiguring]|[ComponentName=mysql;log_error_verbosity=3]|[Description=Reconfiguring the cluster specify component mysql set log_error_verbosity=3] [PASSED]|[SwitchOver]|[ComponentName=mysql]|[Description=SwitchOver the cluster specify component mysql] [PASSED]|[Bench]|[ComponentName=mysql]|[Description=Bench the cluster service with mysql component] [PASSED]|[Bench]|[HostType=LB;ComponentName=mysql]|[Description=Bench the cluster LB service with mysql component] [PASSED]|[Reconfiguring]|[ComponentName=mysql;max_connections=1000,general_log=OFF]|[Description=Reconfiguring the cluster specify component mysql set max_connections=1000,general_log=OFF] [PASSED]|[Reconfiguring]|[ComponentName=mysql;max_connections=2000,general_log=ON,innodb_sort_buffer_size=2097152]|[Description=Reconfiguring the cluster specify component mysql set max_connections=2000,general_log=ON,innodb_sort_buffer_size=2097152] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=mysql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=mysql]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[VerticalScaling]|[ComponentName=mysql]|[Description=VerticalScaling the cluster specify component mysql] [PASSED]|[Failover]|[HA=Network Delay Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[VolumeExpansion]|[ComponentName=mysql]|[Description=VolumeExpansion the cluster specify component mysql] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[HscaleOfflineInstances]|[ComponentName=mysql]|[Description=Hscale the cluster instances offline specify component mysql] [PASSED]|[HscaleOnlineInstances]|[ComponentName=mysql]|[Description=Hscale the cluster instances online specify component mysql] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mysql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=mysql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Full CPU Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=mysql]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[HorizontalScaling Out]|[ComponentName=mysql]|[Description=HorizontalScaling Out the cluster specify component mysql] [PASSED]|[HorizontalScaling In]|[ComponentName=mysql]|[Description=HorizontalScaling In the cluster specify component mysql] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Backup]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Backup] [PASSED]|[Restore]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=xtrabackup]|[Description=Delete the xtrabackup restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=mysql]|[Description=Rebuild the cluster instance specify component mysql] [PASSED]|[Backup]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=xtrabackup]|[Description=Delete the Schedule xtrabackup restore cluster] [PASSED]|[Expose]|[Disable=true;TYPE=intranet;ComponentName=mysql]|[Description=Expose Disable the intranet service with mysql component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]