source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-dmaln ` `kubectl create namespace ns-dmaln` namespace/ns-dmaln created create namespace ns-dmaln done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 60.1M 0 --:--:-- --:--:-- --:--:-- 60.1M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:mysql set component version No resources found no component version found set apecloud-mysql component definition set apecloud-mysql component definition apecloud-mysql not found component version set replicas first:3 set replicas third:3 set minimum cmpv service version set minimum cmpv service version replicas:3 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Not found topology in cluster definition apecloud-mysql LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-dmaln namespace. termination_policy:Delete create 3 replica Delete apecloud-mysql cluster check cluster version check cluster definition check component definition set component definition by component version no component definitions found apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: apemysql-erdpec namespace: ns-dmaln spec: terminationPolicy: Delete componentSpecs: - name: mysql componentDef: apecloud-mysql replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_apemysql-erdpec.yaml` cluster.apps.kubeblocks.io/apemysql-erdpec created apply test_create_apemysql-erdpec.yaml Success `rm -rf test_create_apemysql-erdpec.yaml` check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Jun 19,2025 18:16 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check pod apemysql-erdpec-mysql-0 container_name mysql exist password TR93uUN6pA3234g5 check pod apemysql-erdpec-mysql-0 container_name mysql-exporter exist password TR93uUN6pA3234g5 check pod apemysql-erdpec-mysql-0 container_name lorry exist password TR93uUN6pA3234g5 check pod apemysql-erdpec-mysql-0 container_name config-manager exist password TR93uUN6pA3234g5 No container logs contain secret password. describe cluster `kbcli cluster describe apemysql-erdpec --namespace ns-dmaln ` Name: apemysql-erdpec Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-dmaln Running Delete Endpoints: COMPONENT MODE INTERNAL EXTERNAL mysql ReadWrite apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local:3306 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mysql apemysql-erdpec-mysql-0 leader Running us-west-2a ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 mysql apemysql-erdpec-mysql-1 follower Running us-west-2a ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 mysql apemysql-erdpec-mysql-2 follower Running us-west-2a ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mysql false 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc Images: COMPONENT TYPE IMAGE mysql docker.io/apecloud/apecloud-mysql-server:8.0.30-5.beta3.20231215.ge77d836.14 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-dmaln apemysql-erdpec `kbcli cluster label apemysql-erdpec app.kubernetes.io/instance- --namespace ns-dmaln ` label "app.kubernetes.io/instance" not found. `kbcli cluster label apemysql-erdpec app.kubernetes.io/instance=apemysql-erdpec --namespace ns-dmaln ` `kbcli cluster label apemysql-erdpec --list --namespace ns-dmaln ` NAME NAMESPACE LABELS apemysql-erdpec ns-dmaln app.kubernetes.io/instance=apemysql-erdpec label cluster app.kubernetes.io/instance=apemysql-erdpec Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=apemysql-erdpec --namespace ns-dmaln ` `kbcli cluster label apemysql-erdpec --list --namespace ns-dmaln ` NAME NAMESPACE LABELS apemysql-erdpec ns-dmaln app.kubernetes.io/instance=apemysql-erdpec case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label apemysql-erdpec case.name=kbcli.test2 --overwrite --namespace ns-dmaln ` `kbcli cluster label apemysql-erdpec --list --namespace ns-dmaln ` NAME NAMESPACE LABELS apemysql-erdpec ns-dmaln app.kubernetes.io/instance=apemysql-erdpec case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label apemysql-erdpec case.name- --namespace ns-dmaln ` `kbcli cluster label apemysql-erdpec --list --namespace ns-dmaln ` NAME NAMESPACE LABELS apemysql-erdpec ns-dmaln app.kubernetes.io/instance=apemysql-erdpec delete cluster label case.name Success cluster connect Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-erdpec-mysql-0.apemysql-erdpec-mysql-headless:13306 Leader YES 0 0 2 apemysql-erdpec-mysql-1.apemysql-erdpec-mysql-headless:13306 Follower YES 0 0 3 apemysql-erdpec-mysql-2.apemysql-erdpec-mysql-headless:13306 Follower YES 0 0 `echo 'show variables like "max_connections";' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Variable_name Value max_connections 83 connect cluster Success set max_connections to 83 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-apemysql-erdpec" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-apemysql-erdpec --namespace ns-dmaln ` Error from server (NotFound): pods "test-db-client-executionloop-apemysql-erdpec" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-apemysql-erdpec" not found `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-apemysql-erdpec namespace: ns-dmaln spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local" - "--user" - "root" - "--password" - "TR93uUN6pA3234g5" - "--port" - "3306" - "--dbtype" - "mysql" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-apemysql-erdpec.yaml` pod/test-db-client-executionloop-apemysql-erdpec created apply test-db-client-executionloop-apemysql-erdpec.yaml Success `rm -rf test-db-client-executionloop-apemysql-erdpec.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 1/1 Running 0 57s check pod test-db-client-executionloop-apemysql-erdpec status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-apemysql-erdpec 0/1 Completed 0 63s check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done --host apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local --user root --password TR93uUN6pA3234g5 --port 3306 --dbtype mysql --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:20:51.641 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255), tinyint_col TINYINT, smallint_col SMALLINT, mediumint_col MEDIUMINT, int_col INT, bigint_col BIGINT, float_col FLOAT, double_col DOUBLE, decimal_col DECIMAL(10, 2), date_col DATE, time_col TIME, datetime_col DATETIME, timestamp_col TIMESTAMP, year_col YEAR, char_col CHAR(10), text_col TEXT, blob_col BLOB, binary_col BINARY(10), varbinary_col VARBINARY(255), enum_col ENUM('Option1', 'Option2', 'Option3'), set_col SET('Value1', 'Value2', 'Value3') ); Execution loop start:INSERT INTO executions_loop.executions_loop_table (value, tinyint_col, smallint_col, mediumint_col, int_col, bigint_col, float_col, double_col, decimal_col, date_col, time_col, datetime_col, timestamp_col, year_col, char_col, text_col, blob_col, binary_col, varbinary_col, enum_col, set_col) VALUES ('executions_loop_test_1', 42, 21441, 4742757, 257395157, 7906825453039977951, 0.8771835, 0.09979806133839897, 1.053966300031406, '2025-06-19', '10:20:52', '2025-06-19 10:20:52.34', CURRENT_TIMESTAMP, 1910, '7C5l4M5Nib', 'cweoC5GmFPrvEogDboFHVm3k9Hax9AvcVyIYsdmzakeY0CQorNVbm4r9fkYxZ4olg0oTVyNqHwFR1Fod2LYVuJetPEDjS6WTkLK4aC5CXDuRzjreO91YtFAPyxEsHA2NSZmfR2mtimAYsuFNE2pFMlmZJwWQGWn9DRd5daeKHX5zkGiO4EQBF2GMck7Vj4Qh4qI8h86iXgHtvztNjfgn4VyUp5QAHpOH5dJsW1Lt1j2pqH5W8bQBsTH46LQzIMg', UNHEX(REPLACE('1d1f667e3b00a04a4df2', ' ', '')), UNHEX(REPLACE('6931b8983268092cea29', ' ', '')), UNHEX(REPLACE('aff0cbf94356223adc051de9f1614859c5e12ce819eadfbaa376267e78cda0ea5f8e93b3ae321ba95cb73433562641b5da673d398e5401077656ac4fd76d58d38ebd2894764d20aa93e870b0bf87a2d08b55231af3b0bff55ab258f88f01ac9d855c5d7f480bf6d08b9b5070068af4b90ce12664940ef7e853bf6354eadc42b1ff4d2d6834ee3499e02d3b9a072765344298f139658cb1e738b3df997826c52fae1387ff2e5e827a46b7d0e3248be76a492336227c2954fe368b20c58b5b6257b6389da88e01ea4201b877e4c2614eacefd69d459325651c4b22e636bce4f368308c0a6d90a5d9846c178292ca31b38e3617e6a487f18d53cb9742fc64ba2d', ' ', '')), 'Option2', 'Value3' ); [ 1s ] executions total: 14 successful: 14 failed: 0 disconnect: 0 [ 2s ] executions total: 35 successful: 35 failed: 0 disconnect: 0 [ 3s ] executions total: 58 successful: 58 failed: 0 disconnect: 0 [ 4s ] executions total: 82 successful: 82 failed: 0 disconnect: 0 [ 5s ] executions total: 105 successful: 105 failed: 0 disconnect: 0 [ 6s ] executions total: 129 successful: 129 failed: 0 disconnect: 0 [ 7s ] executions total: 152 successful: 152 failed: 0 disconnect: 0 [ 8s ] executions total: 176 successful: 176 failed: 0 disconnect: 0 [ 9s ] executions total: 197 successful: 197 failed: 0 disconnect: 0 [ 10s ] executions total: 216 successful: 216 failed: 0 disconnect: 0 [ 11s ] executions total: 239 successful: 239 failed: 0 disconnect: 0 [ 12s ] executions total: 265 successful: 265 failed: 0 disconnect: 0 [ 13s ] executions total: 291 successful: 291 failed: 0 disconnect: 0 [ 14s ] executions total: 315 successful: 315 failed: 0 disconnect: 0 [ 15s ] executions total: 336 successful: 336 failed: 0 disconnect: 0 [ 16s ] executions total: 361 successful: 361 failed: 0 disconnect: 0 [ 17s ] executions total: 384 successful: 384 failed: 0 disconnect: 0 [ 18s ] executions total: 410 successful: 410 failed: 0 disconnect: 0 [ 19s ] executions total: 434 successful: 434 failed: 0 disconnect: 0 [ 20s ] executions total: 456 successful: 456 failed: 0 disconnect: 0 [ 21s ] executions total: 485 successful: 485 failed: 0 disconnect: 0 [ 22s ] executions total: 507 successful: 507 failed: 0 disconnect: 0 [ 23s ] executions total: 533 successful: 533 failed: 0 disconnect: 0 [ 24s ] executions total: 558 successful: 558 failed: 0 disconnect: 0 [ 25s ] executions total: 582 successful: 582 failed: 0 disconnect: 0 [ 26s ] executions total: 605 successful: 605 failed: 0 disconnect: 0 [ 27s ] executions total: 632 successful: 632 failed: 0 disconnect: 0 [ 28s ] executions total: 658 successful: 658 failed: 0 disconnect: 0 [ 29s ] executions total: 678 successful: 678 failed: 0 disconnect: 0 [ 30s ] executions total: 704 successful: 704 failed: 0 disconnect: 0 [ 31s ] executions total: 730 successful: 730 failed: 0 disconnect: 0 [ 32s ] executions total: 755 successful: 755 failed: 0 disconnect: 0 [ 33s ] executions total: 781 successful: 781 failed: 0 disconnect: 0 [ 34s ] executions total: 807 successful: 807 failed: 0 disconnect: 0 [ 35s ] executions total: 835 successful: 835 failed: 0 disconnect: 0 [ 36s ] executions total: 860 successful: 860 failed: 0 disconnect: 0 [ 37s ] executions total: 884 successful: 884 failed: 0 disconnect: 0 [ 38s ] executions total: 909 successful: 909 failed: 0 disconnect: 0 [ 39s ] executions total: 932 successful: 932 failed: 0 disconnect: 0 [ 40s ] executions total: 958 successful: 958 failed: 0 disconnect: 0 [ 41s ] executions total: 983 successful: 983 failed: 0 disconnect: 0 [ 42s ] executions total: 1008 successful: 1008 failed: 0 disconnect: 0 [ 43s ] executions total: 1033 successful: 1033 failed: 0 disconnect: 0 [ 44s ] executions total: 1058 successful: 1058 failed: 0 disconnect: 0 [ 45s ] executions total: 1084 successful: 1084 failed: 0 disconnect: 0 [ 46s ] executions total: 1109 successful: 1109 failed: 0 disconnect: 0 [ 47s ] executions total: 1132 successful: 1132 failed: 0 disconnect: 0 [ 48s ] executions total: 1154 successful: 1154 failed: 0 disconnect: 0 [ 49s ] executions total: 1174 successful: 1174 failed: 0 disconnect: 0 [ 50s ] executions total: 1198 successful: 1198 failed: 0 disconnect: 0 [ 51s ] executions total: 1223 successful: 1223 failed: 0 disconnect: 0 [ 52s ] executions total: 1249 successful: 1249 failed: 0 disconnect: 0 [ 53s ] executions total: 1275 successful: 1275 failed: 0 disconnect: 0 [ 54s ] executions total: 1297 successful: 1297 failed: 0 disconnect: 0 [ 55s ] executions total: 1321 successful: 1321 failed: 0 disconnect: 0 [ 56s ] executions total: 1344 successful: 1344 failed: 0 disconnect: 0 [ 57s ] executions total: 1366 successful: 1366 failed: 0 disconnect: 0 [ 58s ] executions total: 1392 successful: 1392 failed: 0 disconnect: 0 [ 60s ] executions total: 1407 successful: 1407 failed: 0 disconnect: 0 Test Result: Total Executions: 1407 Successful Executions: 1407 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: mysql Host: apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local Port: 3306 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 1407 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-apemysql-erdpec --namespace ns-dmaln ` pod/test-db-client-executionloop-apemysql-erdpec patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-apemysql-erdpec" force deleted `echo "CREATE DATABASE IF NOT EXISTS mydb; use mydb; DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); INSERT INTO tmp_table (value) VALUES ('iuktg');" | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. add consistent data iuktg Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"CREATE DATABASE IF NOT EXISTS mydb; use mydb; DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); INSERT INTO tmp_table (value) VALUES ('iuktg');\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. ERROR 7504 (HY000) at line 1: The consensus follower is not allowed to to do current operation. command terminated with exit code 1 check add consistent data readonly Success cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose apemysql-erdpec --auto-approve --force=true --type internet --enable true --components mysql --namespace ns-dmaln ` OpsRequest apemysql-erdpec-expose-rtn82 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-expose-rtn82 -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-expose-rtn82 ns-dmaln Expose apemysql-erdpec mysql Running 0/1 Jun 19,2025 18:22 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-expose-rtn82 ns-dmaln Expose apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 18:22 UTC+0800 check ops status done ops_status:apemysql-erdpec-expose-rtn82 ns-dmaln Expose apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 18:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-expose-rtn82 --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-expose-rtn82 patched `kbcli cluster delete-ops --name apemysql-erdpec-expose-rtn82 --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-expose-rtn82 deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-erdpec" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 action: partition mode: all target: mode: all selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-apemysql-erdpec.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-apemysql-erdpec created apply test-chaos-mesh-networkpartition-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-networkpartition-apemysql-erdpec.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-erdpec" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-0 failover networkpartition Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-erdpec" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-apemysql-erdpec.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-apemysql-erdpec created apply test-chaos-mesh-podfailure-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-podfailure-apemysql-erdpec.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-erdpec" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-2 failover podfailure Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-erdpec" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-2 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-apemysql-erdpec.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-apemysql-erdpec created apply test-chaos-mesh-timeoffset-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-timeoffset-apemysql-erdpec.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-erdpec" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-2 failover timeoffset Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in apemysql-erdpec namespace. `kbcli cluster hscale apemysql-erdpec --auto-approve --force=true --components mysql --replicas 4 --namespace ns-dmaln ` OpsRequest apemysql-erdpec-horizontalscaling-q8ptq created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-horizontalscaling-q8ptq -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-horizontalscaling-q8ptq ns-dmaln HorizontalScaling apemysql-erdpec mysql Running 0/1 Jun 19,2025 18:31 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 apemysql-erdpec-mysql-3 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:32 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-3 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done No resources found in apemysql-erdpec namespace. check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-horizontalscaling-q8ptq ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 18:31 UTC+0800 check ops status done ops_status:apemysql-erdpec-horizontalscaling-q8ptq ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 18:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-horizontalscaling-q8ptq --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-horizontalscaling-q8ptq patched `kbcli cluster delete-ops --name apemysql-erdpec-horizontalscaling-q8ptq --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-horizontalscaling-q8ptq deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-3(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in apemysql-erdpec namespace. `kbcli cluster hscale apemysql-erdpec --auto-approve --force=true --components mysql --replicas 3 --namespace ns-dmaln ` OpsRequest apemysql-erdpec-horizontalscaling-544wj created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-horizontalscaling-544wj -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-horizontalscaling-544wj ns-dmaln HorizontalScaling apemysql-erdpec mysql Running 0/1 Jun 19,2025 18:33 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done No resources found in apemysql-erdpec namespace. check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-horizontalscaling-544wj ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 18:33 UTC+0800 check ops status done ops_status:apemysql-erdpec-horizontalscaling-544wj ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 18:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-horizontalscaling-544wj --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-horizontalscaling-544wj patched `kbcli cluster delete-ops --name apemysql-erdpec-horizontalscaling-544wj --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-horizontalscaling-544wj deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` connect checking... connect checking... check cluster connect done check failover pod name failover pod name:apemysql-erdpec-mysql-1 failover kill1 Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-erdpec" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-apemysql-erdpec.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-apemysql-erdpec created apply test-chaos-mesh-networkbandwidthover-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-apemysql-erdpec.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 18:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-apemysql-erdpec" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-apemysql-erdpec patched check failover pod name failover pod name:apemysql-erdpec-mysql-0 failover networkbandwidthover Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart apemysql-erdpec --auto-approve --force=true --namespace ns-dmaln ` OpsRequest apemysql-erdpec-restart-vwltc created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-restart-vwltc -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-restart-vwltc ns-dmaln Restart apemysql-erdpec mysql Running 0/3 Jun 19,2025 18:38 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:41 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-restart-vwltc ns-dmaln Restart apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 18:38 UTC+0800 check ops status done ops_status:apemysql-erdpec-restart-vwltc ns-dmaln Restart apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 18:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-restart-vwltc --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-restart-vwltc patched `kbcli cluster delete-ops --name apemysql-erdpec-restart-vwltc --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-restart-vwltc deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-erdpec" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-2 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-apemysql-erdpec.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-apemysql-erdpec created apply test-chaos-mesh-networkduplicate-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-networkduplicate-apemysql-erdpec.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:41 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-erdpec" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-2 failover networkduplicate Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-erdpec" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-2 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-apemysql-erdpec.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-apemysql-erdpec created apply test-chaos-mesh-networklossover-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-networklossover-apemysql-erdpec.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:41 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-apemysql-erdpec" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-apemysql-erdpec patched check failover pod name failover pod name:apemysql-erdpec-mysql-0 failover networklossover Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-erdpec" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-apemysql-erdpec.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-apemysql-erdpec created apply test-chaos-mesh-dnsrandom-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-dnsrandom-apemysql-erdpec.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:41 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-erdpec" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-0 failover dnsrandom Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-erdpec" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-apemysql-erdpec.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-apemysql-erdpec created apply test-chaos-mesh-oom-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-oom-apemysql-erdpec.yaml` check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:41 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org/test-chaos-mesh-oom-apemysql-erdpec patched check failover pod name stresschaos.chaos-mesh.org "test-chaos-mesh-oom-apemysql-erdpec" force deleted failover pod name:apemysql-erdpec-mysql-0 failover oom Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover fullcpuover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-erdpec" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-apemysql-erdpec.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-apemysql-erdpec created apply test-chaos-mesh-fullcpuover-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-fullcpuover-apemysql-erdpec.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:41 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-erdpec" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-0 failover fullcpuover Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod apemysql-erdpec-mysql-0 --namespace ns-dmaln -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-10-206.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod apemysql-erdpec-mysql-1 --namespace ns-dmaln -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-5-129.us-west-2.compute.internal success kubectl get pod apemysql-erdpec-mysql-2 --namespace ns-dmaln -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-14-205.us-west-2.compute.internal success kubectl drain ip-172-31-10-206.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-10-206.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-2clvr, kb-mqtcr/kb-addon-apecloud-otel-collector-sg5pq, kube-system/aws-node-rws5g, kube-system/ebs-csi-node-bn65b, kube-system/kube-proxy-rfmmk evicting pod ns-qgdgh/gaussdb-ofzmam-gaussdb-1 evicting pod ns-dmaln/apemysql-erdpec-mysql-0 pod/apemysql-erdpec-mysql-0 evicted pod/gaussdb-ofzmam-gaussdb-1 evicted node/ip-172-31-10-206.us-west-2.compute.internal drained kubectl uncordon ip-172-31-10-206.us-west-2.compute.internal node/ip-172-31-10-206.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:53 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 18:40 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check failover pod name failover pod name:apemysql-erdpec-mysql-2 failover drainnode Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale apemysql-erdpec --auto-approve --force=true --components mysql --cpu 200m --memory 0.6Gi --namespace ns-dmaln ` OpsRequest apemysql-erdpec-verticalscaling-cqpzt created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-verticalscaling-cqpzt -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-verticalscaling-cqpzt ns-dmaln VerticalScaling apemysql-erdpec mysql Running 0/3 Jun 19,2025 18:55 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 18:56 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 18:55 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:57 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-verticalscaling-cqpzt ns-dmaln VerticalScaling apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 18:55 UTC+0800 check ops status done ops_status:apemysql-erdpec-verticalscaling-cqpzt ns-dmaln VerticalScaling apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 18:55 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-verticalscaling-cqpzt --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-verticalscaling-cqpzt patched `kbcli cluster delete-ops --name apemysql-erdpec-verticalscaling-cqpzt --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-verticalscaling-cqpzt deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success skip cluster Upgrade cluster configure component_tmp: mysql check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure apemysql-erdpec --auto-approve --force=true --set consensus_log_level=LOG_INFO,log_error_verbosity=3 --components mysql --config-spec mysql-consensusset-config --config-file my.cnf --namespace ns-dmaln ` Will updated configure file meta: ConfigSpec: mysql-consensusset-config ConfigFile: my.cnf ComponentName: mysql ClusterName: apemysql-erdpec OpsRequest apemysql-erdpec-reconfiguring-qnms8 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-reconfiguring-qnms8 -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-reconfiguring-qnms8 ns-dmaln Reconfiguring apemysql-erdpec mysql Running -/- Jun 19,2025 18:58 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:59 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:00 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-reconfiguring-qnms8 ns-dmaln Reconfiguring apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 18:58 UTC+0800 check ops status done ops_status:apemysql-erdpec-reconfiguring-qnms8 ns-dmaln Reconfiguring apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 18:58 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-reconfiguring-qnms8 --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-reconfiguring-qnms8 patched `kbcli cluster delete-ops --name apemysql-erdpec-reconfiguring-qnms8 --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-reconfiguring-qnms8 deleted check config variables Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: LOG_INFO configure:[consensus_log_level] result actual:[LOG_INFO] equal expected:[LOG_INFO] Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: 3 configure:[log_error_verbosity] result actual:[3] equal expected:[3] `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote apemysql-erdpec --auto-approve --force=true --component mysql --namespace ns-dmaln ` OpsRequest apemysql-erdpec-custom-crxt9 created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-custom-crxt9 -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-custom-crxt9 ns-dmaln Custom apemysql-erdpec mysql Running 0/1 Jun 19,2025 19:01 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:59 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:00 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-custom-crxt9 ns-dmaln Custom apemysql-erdpec mysql Failed 1/1 Jun 19,2025 19:01 UTC+0800 check ops status done --------------------------------------get cluster apemysql-erdpec yaml-------------------------------------- `kubectl get cluster apemysql-erdpec -o yaml --namespace ns-dmaln ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-06-19T10:40:55.565516506Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"apemysql-erdpec","namespace":"ns-dmaln"***,"spec":***"componentSpecs":[***"componentDef":"apecloud-mysql","name":"mysql","replicas":3,"resources":***"limits":***"cpu":"100m","memory":"0.5Gi"***,"requests":***"cpu":"100m","memory":"0.5Gi"***,"volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"1Gi"***,"storageClassName":null***]***],"terminationPolicy":"Delete"*** creationTimestamp: "2025-06-19T10:16:25Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 8 labels: app.kubernetes.io/instance: apemysql-erdpec name: apemysql-erdpec namespace: ns-dmaln resourceVersion: "68482" uid: b80a8d6b-8020-4dd3-9818-a27aaf637d9e spec: componentSpecs: - componentDef: apecloud-mysql name: mysql replicas: 3 resources: limits: cpu: 200m memory: 644245094400m requests: cpu: 200m memory: 644245094400m serviceVersion: 8.0.30 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/aws-load-balancer-internal: "false" service.beta.kubernetes.io/aws-load-balancer-type: nlb componentSelector: mysql name: mysql-internet roleSelector: leader serviceName: mysql-internet spec: ports: - name: mysql nodePort: 30732 port: 3306 protocol: TCP targetPort: mysql - name: paxos nodePort: 31583 port: 13306 protocol: TCP targetPort: paxos type: LoadBalancer storage: size: "0" terminationPolicy: Delete status: components: mysql: phase: Running podsReady: true podsReadyTime: "2025-06-19T11:01:20Z" conditions: - lastTransitionTime: "2025-06-19T10:16:25Z" message: 'The operator has started the provisioning of Cluster: apemysql-erdpec' observedGeneration: 8 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-06-19T10:16:25Z" message: Successfully applied for resources observedGeneration: 8 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-06-19T11:01:20Z" message: all pods of components are ready, waiting for the probe detection successful reason: AllReplicasReady status: "True" type: ReplicasReady - lastTransitionTime: "2025-06-19T11:01:20Z" message: 'Cluster: apemysql-erdpec is ready, current phase is Running' reason: ClusterReady status: "True" type: Ready observedGeneration: 8 phase: Running ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster apemysql-erdpec-------------------------------------- `kubectl describe cluster apemysql-erdpec --namespace ns-dmaln ` Name: apemysql-erdpec Namespace: ns-dmaln Labels: app.kubernetes.io/instance=apemysql-erdpec Annotations: kubeblocks.io/reconcile: 2025-06-19T10:40:55.565516506Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-06-19T10:16:25Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 8 Resource Version: 68482 UID: b80a8d6b-8020-4dd3-9818-a27aaf637d9e Spec: Component Specs: Component Def: apecloud-mysql Name: mysql Replicas: 3 Resources: Limits: Cpu: 200m Memory: 644245094400m Requests: Cpu: 200m Memory: 644245094400m Service Version: 8.0.30 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 1Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/aws-load-balancer-internal: false service.beta.kubernetes.io/aws-load-balancer-type: nlb Component Selector: mysql Name: mysql-internet Role Selector: leader Service Name: mysql-internet Spec: Ports: Name: mysql Node Port: 30732 Port: 3306 Protocol: TCP Target Port: mysql Name: paxos Node Port: 31583 Port: 13306 Protocol: TCP Target Port: paxos Type: LoadBalancer Storage: Size: 0 Termination Policy: Delete Status: Components: Mysql: Phase: Running Pods Ready: true Pods Ready Time: 2025-06-19T11:01:20Z Conditions: Last Transition Time: 2025-06-19T10:16:25Z Message: The operator has started the provisioning of Cluster: apemysql-erdpec Observed Generation: 8 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-06-19T10:16:25Z Message: Successfully applied for resources Observed Generation: 8 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-06-19T11:01:20Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2025-06-19T11:01:20Z Message: Cluster: apemysql-erdpec is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 8 Phase: Running Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 45m cluster-controller component is Creating Normal PreCheckSucceed 39m (x5 over 45m) cluster-controller The operator has started the provisioning of Cluster: apemysql-erdpec Normal ApplyResourcesSucceed 39m (x5 over 45m) cluster-controller Successfully applied for resources Warning ComponentsNotReady 39m cluster-controller pods are unavailable in Components: [mysql], refer to related component message in Cluster.status.components Warning Abnormal 39m (x2 over 42m) cluster-controller Cluster: apemysql-erdpec is Abnormal, check according to the components message Warning ReplicasNotReady 39m cluster-controller pods are not ready in Components: [mysql], refer to related component message in Cluster.status.components Normal AllReplicasReady 37m (x2 over 42m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 37m (x2 over 42m) cluster-controller Cluster: apemysql-erdpec is ready, current phase is Running Normal Running 37m (x2 over 42m) cluster-controller Cluster: apemysql-erdpec is ready, current phase is Running Normal ComponentPhaseTransition 34m (x2 over 36m) cluster-controller component is Updating Normal HorizontalScale 30m (x11 over 31m) component-controller start horizontal scale component mysql of cluster apemysql-erdpec from 3 to 4 Normal HorizontalScale 28m component-controller start horizontal scale component mysql of cluster apemysql-erdpec from 4 to 3 Warning Unhealthy 21m event-controller Pod apemysql-erdpec-mysql-0: Readiness probe failed: Get "http://172.31.15.113:3501/v1.0/checkrole": dial tcp 172.31.15.113:3501: connect: connection refused Normal ComponentPhaseTransition 10m (x4 over 42m) cluster-controller component is Abnormal Normal ComponentPhaseTransition 4m10s (x20 over 42m) cluster-controller component is Running ------------------------------------------------------------------------------------------------------------------ ops_status:apemysql-erdpec-custom-crxt9 ns-dmaln Custom apemysql-erdpec mysql Failed 1/1 Jun 19,2025 19:01 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-custom-crxt9 --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-custom-crxt9 patched `kbcli cluster delete-ops --name apemysql-erdpec-custom-crxt9 --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-custom-crxt9 deleted switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... switchover pod:apemysql-erdpec-mysql-0 checking switchover... check switchover timeout `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-apemysql-erdpec namespace: ns-dmaln spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local,user:root,password:TR93uUN6pA3234g5,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-apemysql-erdpec.yaml` pod/benchtest-apemysql-erdpec created apply benchtest-apemysql-erdpec.yaml Success `rm -rf benchtest-apemysql-erdpec.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 23s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 29s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 1/1 Running 0 46s check pod benchtest-apemysql-erdpec status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 Completed 0 52s `kubectl logs benchtest-apemysql-erdpec --tail 30 --namespace ns-dmaln ` [ 7s ] thds: 4 tps: 30.04 qps: 618.77 (r/w/o: 432.54/126.16/60.07) lat (ms,99%): 193.38 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 31.01 qps: 581.14 (r/w/o: 409.10/110.03/62.01) lat (ms,99%): 200.47 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 28.00 qps: 587.93 (r/w/o: 409.95/121.99/55.99) lat (ms,99%): 287.38 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 25.00 qps: 499.02 (r/w/o: 351.01/98.00/50.00) lat (ms,99%): 204.11 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 3976 write: 1136 other: 568 total: 5680 transactions: 284 (28.35 per sec.) queries: 5680 (566.98 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0165s total number of events: 284 Latency (ms): min: 17.70 avg: 140.98 max: 288.76 99th percentile: 204.11 sum: 40039.45 Threads fairness: events (avg/stddev): 71.0000/0.71 execution time (avg/stddev): 10.0099/0.00 `kubectl delete pod benchtest-apemysql-erdpec --force --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-apemysql-erdpec" force deleted No resources found in ns-dmaln namespace. `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: benchtest-apemysql-erdpec namespace: ns-dmaln spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:ab34dff3047d34c46bab2e772224f6fa-5d2afde21a93c3e5.elb.us-west-2.amazonaws.com,user:root,password:TR93uUN6pA3234g5,port:3306,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-apemysql-erdpec.yaml` pod/benchtest-apemysql-erdpec created apply benchtest-apemysql-erdpec.yaml Success `rm -rf benchtest-apemysql-erdpec.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 1/1 Running 0 12s check pod benchtest-apemysql-erdpec status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-apemysql-erdpec 0/1 Completed 0 17s `kubectl logs benchtest-apemysql-erdpec --tail 30 --namespace ns-dmaln ` [ 7s ] thds: 4 tps: 26.00 qps: 536.00 (r/w/o: 376.00/108.00/52.00) lat (ms,99%): 204.11 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 27.00 qps: 533.90 (r/w/o: 379.93/99.98/53.99) lat (ms,99%): 204.11 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 26.00 qps: 506.05 (r/w/o: 347.03/108.01/51.00) lat (ms,99%): 207.82 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 25.99 qps: 534.88 (r/w/o: 377.92/103.98/52.99) lat (ms,99%): 196.89 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 3780 write: 1080 other: 540 total: 5400 transactions: 270 (26.70 per sec.) queries: 5400 (533.91 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.1124s total number of events: 270 Latency (ms): min: 93.01 avg: 149.56 max: 287.69 99th percentile: 207.82 sum: 40381.80 Threads fairness: events (avg/stddev): 67.5000/2.06 execution time (avg/stddev): 10.0955/0.01 `kubectl delete pod benchtest-apemysql-erdpec --force --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-apemysql-erdpec" force deleted test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-erdpec" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-apemysql-erdpec.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-apemysql-erdpec created apply test-chaos-mesh-dnserror-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-dnserror-apemysql-erdpec.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:59 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:00 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-erdpec" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-0 failover dnserror Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-erdpec" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-apemysql-erdpec.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-apemysql-erdpec created apply test-chaos-mesh-networkcorruptover-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-apemysql-erdpec.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:59 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:00 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-erdpec" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-2 failover networkcorruptover Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster configure component_tmp: mysql check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure apemysql-erdpec --auto-approve --force=true --set max_connections=1000,general_log=OFF --components mysql --config-spec mysql-consensusset-config --config-file my.cnf --namespace ns-dmaln ` Will updated configure file meta: ConfigSpec: mysql-consensusset-config ConfigFile: my.cnf ComponentName: mysql ClusterName: apemysql-erdpec OpsRequest apemysql-erdpec-reconfiguring-jcdzb created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-reconfiguring-jcdzb -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-reconfiguring-jcdzb ns-dmaln Reconfiguring apemysql-erdpec mysql Running -/- Jun 19,2025 19:14 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:59 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:00 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-reconfiguring-jcdzb ns-dmaln Reconfiguring apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:14 UTC+0800 check ops status done ops_status:apemysql-erdpec-reconfiguring-jcdzb ns-dmaln Reconfiguring apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:14 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-reconfiguring-jcdzb --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-reconfiguring-jcdzb patched `kbcli cluster delete-ops --name apemysql-erdpec-reconfiguring-jcdzb --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-reconfiguring-jcdzb deleted check config variables Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: 1000 configure:[max_connections] result actual:[1000] equal expected:[1000] Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: OFF configure:[general_log] result actual:[OFF] equal expected:[OFF] `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file value iuktg Warning: arning] Using a password on the command line interface can be insecure. check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster configure component_tmp: mysql check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure apemysql-erdpec --auto-approve --force=true --set max_connections=2000,general_log=ON,innodb_sort_buffer_size=2097152 --components mysql --config-spec mysql-consensusset-config --config-file my.cnf --namespace ns-dmaln ` Will updated configure file meta: ConfigSpec: mysql-consensusset-config ConfigFile: my.cnf ComponentName: mysql ClusterName: apemysql-erdpec OpsRequest apemysql-erdpec-reconfiguring-2ptrx created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-reconfiguring-2ptrx -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-reconfiguring-2ptrx ns-dmaln Reconfiguring apemysql-erdpec mysql Running -/- Jun 19,2025 19:15 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-reconfiguring-2ptrx ns-dmaln Reconfiguring apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:15 UTC+0800 check ops status done ops_status:apemysql-erdpec-reconfiguring-2ptrx ns-dmaln Reconfiguring apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:15 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-reconfiguring-2ptrx --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-reconfiguring-2ptrx patched `kbcli cluster delete-ops --name apemysql-erdpec-reconfiguring-2ptrx --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-reconfiguring-2ptrx deleted check config variables Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: 2000 configure:[max_connections] result actual:[2000] equal expected:[2000] Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: ON configure:[general_log] result actual:[ON] equal expected:[ON] Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. current value_actual: 2097152 configure:[innodb_sort_buffer_size] result actual:[2097152] equal expected:[2097152] `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-apemysql-erdpec" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-apemysql-erdpec --namespace ns-dmaln ` Error from server (NotFound): pods "test-db-client-connectionstress-apemysql-erdpec" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-apemysql-erdpec" not found `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-apemysql-erdpec namespace: ns-dmaln spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local" - "--user" - "root" - "--password" - "TR93uUN6pA3234g5" - "--port" - "3306" - "--database" - "mysql" - "--dbtype" - "mysql" - "--test" - "connectionstress" - "--connections" - "83" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-apemysql-erdpec.yaml` pod/test-db-client-connectionstress-apemysql-erdpec created apply test-db-client-connectionstress-apemysql-erdpec.yaml Success `rm -rf test-db-client-connectionstress-apemysql-erdpec.yaml` check pod status check pod test-db-client-connectionstress-apemysql-erdpec status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-apemysql-erdpec 0/1 Completed 0 5s check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:16 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done --host apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local --user root --password TR93uUN6pA3234g5 --port 3306 --database mysql --dbtype mysql --test connectionstress --connections 83 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 11:18:46.518 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Test Result: null Connection Information: Database Type: mysql Host: apemysql-erdpec-mysql.ns-dmaln.svc.cluster.local Port: 3306 Database: mysql Table: User: root Org: Access Mode: mysql Test Type: connectionstress Connection Count: 83 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-apemysql-erdpec --namespace ns-dmaln ` pod/test-db-client-connectionstress-apemysql-erdpec patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-apemysql-erdpec" force deleted check failover pod name failover pod name:apemysql-erdpec-mysql-1 failover connectionstress Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster hscale offline instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-erdpec-hscaleoffinstance- labels: app.kubernetes.io/instance: apemysql-erdpec app.kubernetes.io/managed-by: kubeblocks namespace: ns-dmaln spec: type: HorizontalScaling clusterName: apemysql-erdpec force: true horizontalScaling: - componentName: mysql scaleIn: onlineInstancesToOffline: - apemysql-erdpec-mysql-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-erdpec.yaml` opsrequest.apps.kubeblocks.io/apemysql-erdpec-hscaleoffinstance-86gpc created create test_ops_cluster_apemysql-erdpec.yaml Success `rm -rf test_ops_cluster_apemysql-erdpec.yaml` check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-hscaleoffinstance-86gpc ns-dmaln HorizontalScaling apemysql-erdpec mysql Running 0/1 Jun 19,2025 19:19 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-hscaleoffinstance-86gpc ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 19:19 UTC+0800 check ops status done ops_status:apemysql-erdpec-hscaleoffinstance-86gpc ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 19:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-hscaleoffinstance-86gpc --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-hscaleoffinstance-86gpc patched `kbcli cluster delete-ops --name apemysql-erdpec-hscaleoffinstance-86gpc --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-hscaleoffinstance-86gpc deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster hscale online instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-erdpec-hscaleoninstance- labels: app.kubernetes.io/instance: apemysql-erdpec app.kubernetes.io/managed-by: kubeblocks namespace: ns-dmaln spec: type: HorizontalScaling clusterName: apemysql-erdpec force: true horizontalScaling: - componentName: mysql scaleOut: offlineInstancesToOnline: - apemysql-erdpec-mysql-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-erdpec.yaml` opsrequest.apps.kubeblocks.io/apemysql-erdpec-hscaleoninstance-54n9l created create test_ops_cluster_apemysql-erdpec.yaml Success `rm -rf test_ops_cluster_apemysql-erdpec.yaml` check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-hscaleoninstance-54n9l ns-dmaln HorizontalScaling apemysql-erdpec mysql Running 0/1 Jun 19,2025 19:20 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:21 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:16 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:17 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-hscaleoninstance-54n9l ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 19:20 UTC+0800 check ops status done ops_status:apemysql-erdpec-hscaleoninstance-54n9l ns-dmaln HorizontalScaling apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 19:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-hscaleoninstance-54n9l --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-hscaleoninstance-54n9l patched `kbcli cluster delete-ops --name apemysql-erdpec-hscaleoninstance-54n9l --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-hscaleoninstance-54n9l deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop apemysql-erdpec --auto-approve --force=true --namespace ns-dmaln ` OpsRequest apemysql-erdpec-stop-gwfgj created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-stop-gwfgj -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-stop-gwfgj ns-dmaln Stop apemysql-erdpec mysql Running 0/3 Jun 19,2025 19:22 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Stopping Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-stop-gwfgj ns-dmaln Stop apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:22 UTC+0800 check ops status done ops_status:apemysql-erdpec-stop-gwfgj ns-dmaln Stop apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-stop-gwfgj --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-stop-gwfgj patched `kbcli cluster delete-ops --name apemysql-erdpec-stop-gwfgj --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-stop-gwfgj deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start apemysql-erdpec --force=true --namespace ns-dmaln ` OpsRequest apemysql-erdpec-start-vf8tx created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-start-vf8tx -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-start-vf8tx ns-dmaln Start apemysql-erdpec mysql Running 0/3 Jun 19,2025 19:24 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-start-vf8tx ns-dmaln Start apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:24 UTC+0800 check ops status done ops_status:apemysql-erdpec-start-vf8tx ns-dmaln Start apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:24 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-start-vf8tx --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-start-vf8tx patched `kbcli cluster delete-ops --name apemysql-erdpec-start-vf8tx --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-start-vf8tx deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-0: out of apemysql-erdpec-mysql-0(leader), apemysql-erdpec-mysql-1(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster update monitor KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster apemysql-erdpec --namespace ns-dmaln --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/apemysql-erdpec patched check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done create curl-jq pod apemysql-erdpec-kb-curl `kubectl run --namespace kb-mqtcr apemysql-erdpec-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/apemysql-erdpec-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE apemysql-erdpec-kb-curl 0/1 ContainerCreating 0 1s check pod apemysql-erdpec-kb-curl status done `kubectl wait --for=condition=Ready pod apemysql-erdpec-kb-curl --namespace kb-mqtcr` pod/apemysql-erdpec-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE apemysql-erdpec-kb-curl 1/1 Running 0 6s kubectl exec -it apemysql-erdpec-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it apemysql-erdpec-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 121 100 121 0 0 38038 0 --:--:-- --:--:-- --:--:-- 40333 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750332380.397,"12.79999999999999"]***]*** k8s_pod_cpu_limit: 1750332380.397 monitor cluster Success `kubectl delete pod apemysql-erdpec-kb-curl --namespace kb-mqtcr --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "apemysql-erdpec-kb-curl" force deleted check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-0;follower: apemysql-erdpec-mysql-1 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done test failover networkdelayover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelayover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-erdpec" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-erdpec" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelayover-apemysql-erdpec namespace: ns-dmaln spec: selector: namespaces: - ns-dmaln labelSelectors: apps.kubeblocks.io/pod-name: apemysql-erdpec-mysql-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelayover-apemysql-erdpec.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelayover-apemysql-erdpec created apply test-chaos-mesh-networkdelayover-apemysql-erdpec.yaml Success `rm -rf test-chaos-mesh-networkdelayover-apemysql-erdpec.yaml` networkdelayover chaos test waiting 120 seconds check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-3-33.us-west-2.compute.internal/172.31.3.33 Jun 19,2025 19:25 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-2;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-1 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelayover-apemysql-erdpec --namespace ns-dmaln ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-erdpec" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelayover-apemysql-erdpec" not found check failover pod name failover pod name:apemysql-erdpec-mysql-2 failover networkdelayover Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-2: out of apemysql-erdpec-mysql-2(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-1(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-2 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success test failover delete pod:apemysql-erdpec-mysql-2 `kubectl delete pod apemysql-erdpec-mysql-2 --namespace ns-dmaln ` pod "apemysql-erdpec-mysql-2" deleted check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check failover pod name failover pod name:apemysql-erdpec-mysql-1 failover Success `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster update enable-all-logs `kbcli cluster update apemysql-erdpec --enable-all-logs=true --namespace ns-dmaln ` error: name is required check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl get pvc -l app.kubernetes.io/instance=apemysql-erdpec,apps.kubeblocks.io/component-name=mysql,apps.kubeblocks.io/vct-name=data --namespace ns-dmaln ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in apemysql-erdpec namespace. `kbcli cluster volume-expand apemysql-erdpec --auto-approve --force=true --components mysql --volume-claim-templates data --storage 5Gi --namespace ns-dmaln ` OpsRequest apemysql-erdpec-volumeexpansion-qbqsd created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-volumeexpansion-qbqsd -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-volumeexpansion-qbqsd ns-dmaln VolumeExpansion apemysql-erdpec mysql Running 0/3 Jun 19,2025 19:30 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done No resources found in apemysql-erdpec namespace. check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-volumeexpansion-qbqsd ns-dmaln VolumeExpansion apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:30 UTC+0800 check ops status done ops_status:apemysql-erdpec-volumeexpansion-qbqsd ns-dmaln VolumeExpansion apemysql-erdpec mysql Succeed 3/3 Jun 19,2025 19:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-volumeexpansion-qbqsd --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-volumeexpansion-qbqsd patched `kbcli cluster delete-ops --name apemysql-erdpec-volumeexpansion-qbqsd --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-volumeexpansion-qbqsd deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update apemysql-erdpec --termination-policy=WipeOut --namespace ns-dmaln ` cluster.apps.kubeblocks.io/apemysql-erdpec updated check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `kubectl get backupschedule -l app.kubernetes.io/instance=apemysql-erdpec ` `kubectl get backupschedule apemysql-erdpec-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched check backup status `kbcli cluster list-backups apemysql-erdpec --namespace ns-dmaln ` No backups found in ns-dmaln namespace. No backups found in ns-dmaln namespace. backup_status:-- No backups found in ns-dmaln namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=apemysql-erdpec ` `kubectl get backupschedule apemysql-erdpec-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched (no change) backup_status:apemysql-erdpec-xtrabackup-Running backup_status:apemysql-erdpec-xtrabackup-Running check backup status done backup_status:apemysql-erdpec-xtrabackup-20250619113300 ns-dmaln apemysql-erdpec xtrabackup Completed 7979269 16s Jun 19,2025 19:33 UTC+0800 Jun 19,2025 19:33 UTC+0800 Jun 26,2025 19:33 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=apemysql-erdpec ` `kubectl get backupschedule apemysql-erdpec-mysql-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/apemysql-erdpec-mysql-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "apemysql-erdpec-backup" not found `kbcli cluster describe-backup apemysql-erdpec-xtrabackup-20250619113300 --namespace ns-dmaln ` Name: apemysql-erdpec-xtrabackup-20250619113300 Cluster: apemysql-erdpec Namespace: ns-dmaln Spec: Method: xtrabackup Policy Name: apemysql-erdpec-mysql-backup-policy Status: Phase: Completed Total Size: 7979269 ActionSet Name: xtrabackup-for-apecloud-mysql Repository: backuprepo-kbcli-test Duration: 16s Expiration Time: Jun 26,2025 19:33 UTC+0800 Start Time: Jun 19,2025 19:33 UTC+0800 Completion Time: Jun 19,2025 19:33 UTC+0800 Path: /ns-dmaln/apemysql-erdpec-b80a8d6b-8020-4dd3-9818-a27aaf637d9e/mysql/apemysql-erdpec-xtrabackup-20250619113300 Time Range Start: Jun 19,2025 19:33 UTC+0800 Time Range End: Jun 19,2025 19:33 UTC+0800 Warning Events: `kbcli cluster restore apemysql-erdpec-backup --backup apemysql-erdpec-xtrabackup-20250619113300 --namespace ns-dmaln ` Cluster apemysql-erdpec-backup created check cluster status `kbcli cluster list apemysql-erdpec-backup --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec-backup ns-dmaln WipeOut Jun 19,2025 19:33 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec-backup --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-backup-mysql-0 ns-dmaln apemysql-erdpec-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 19:34 UTC+0800 apemysql-erdpec-backup-mysql-1 ns-dmaln apemysql-erdpec-backup mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-97.us-west-2.compute.internal/172.31.1.97 Jun 19,2025 19:34 UTC+0800 apemysql-erdpec-backup-mysql-2 ns-dmaln apemysql-erdpec-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:34 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-backup-mysql-1;follower: apemysql-erdpec-backup-mysql-0 apemysql-erdpec-backup-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec-backup --namespace ns-dmaln` check cluster connect done `kbcli cluster describe-backup apemysql-erdpec-xtrabackup-20250619113300 --namespace ns-dmaln ` Name: apemysql-erdpec-xtrabackup-20250619113300 Cluster: apemysql-erdpec Namespace: ns-dmaln Spec: Method: xtrabackup Policy Name: apemysql-erdpec-mysql-backup-policy Status: Phase: Completed Total Size: 7979269 ActionSet Name: xtrabackup-for-apecloud-mysql Repository: backuprepo-kbcli-test Duration: 16s Expiration Time: Jun 26,2025 19:33 UTC+0800 Start Time: Jun 19,2025 19:33 UTC+0800 Completion Time: Jun 19,2025 19:33 UTC+0800 Path: /ns-dmaln/apemysql-erdpec-b80a8d6b-8020-4dd3-9818-a27aaf637d9e/mysql/apemysql-erdpec-xtrabackup-20250619113300 Time Range Start: Jun 19,2025 19:33 UTC+0800 Time Range End: Jun 19,2025 19:33 UTC+0800 Warning Events: cluster connect Connect to instance apemysql-erdpec-backup-mysql-1: out of apemysql-erdpec-backup-mysql-1(leader), apemysql-erdpec-backup-mysql-0(follower), apemysql-erdpec-backup-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-erdpec-backup-mysql-0.apemysql-erdpec-backup-mysql-headless:13306 Follower YES 0 0 2 apemysql-erdpec-backup-mysql-1.apemysql-erdpec-backup-mysql-headless:13306 Leader YES 0 0 3 apemysql-erdpec-backup-mysql-2.apemysql-erdpec-backup-mysql-headless:13306 Follower YES 0 0 `echo 'show variables like "max_connections";' | kbcli cluster connect apemysql-erdpec-backup --namespace ns-dmaln ` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Connect to instance apemysql-erdpec-backup-mysql-1: out of apemysql-erdpec-backup-mysql-1(leader), apemysql-erdpec-backup-mysql-0(follower), apemysql-erdpec-backup-mysql-2(follower) Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster apemysql-erdpec-backup `kbcli cluster delete apemysql-erdpec-backup --auto-approve --namespace ns-dmaln ` Cluster apemysql-erdpec-backup deleted pod_info:apemysql-erdpec-backup-mysql-0 4/4 Running 0 102s apemysql-erdpec-backup-mysql-1 4/4 Running 0 69s apemysql-erdpec-backup-mysql-2 4/4 Running 0 52s pod_info:apemysql-erdpec-backup-mysql-0 3/4 Terminating 0 2m3s apemysql-erdpec-backup-mysql-1 3/4 Terminating 0 90s apemysql-erdpec-backup-mysql-2 3/4 Terminating 0 73s No resources found in ns-dmaln namespace. delete cluster pod done No resources found in ns-dmaln namespace. check cluster resource non-exist OK: pvc No resources found in ns-dmaln namespace. delete cluster done No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups apemysql-erdpec-xtrabackup-20250619113300 --namespace ns-dmaln ` backup.dataprotection.kubeblocks.io/apemysql-erdpec-xtrabackup-20250619113300 patched `kbcli cluster delete-backup apemysql-erdpec --name apemysql-erdpec-xtrabackup-20250619113300 --force --auto-approve --namespace ns-dmaln ` Backup apemysql-erdpec-xtrabackup-20250619113300 deleted No opsrequests found in ns-dmaln namespace. cluster volume-snapshot backup `kbcli cluster backup apemysql-erdpec --method volume-snapshot --namespace ns-dmaln ` Backup backup-ns-dmaln-apemysql-erdpec-20250619193635 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-dmaln-apemysql-erdpec-20250619193635 -n ns-dmaln check backup status `kbcli cluster list-backups apemysql-erdpec --namespace ns-dmaln ` No backups found in ns-dmaln namespace. backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running backup_status:apemysql-erdpec-volume-snapshot-Running check backup status done backup_status:backup-ns-dmaln-apemysql-erdpec-20250619193635 ns-dmaln apemysql-erdpec volume-snapshot Completed 5Gi 68s Jun 19,2025 19:36 UTC+0800 Jun 19,2025 19:37 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "apemysql-erdpec-backup" not found `kbcli cluster describe-backup backup-ns-dmaln-apemysql-erdpec-20250619193635 --namespace ns-dmaln ` Name: backup-ns-dmaln-apemysql-erdpec-20250619193635 Cluster: apemysql-erdpec Namespace: ns-dmaln Spec: Method: volume-snapshot Policy Name: apemysql-erdpec-mysql-backup-policy Status: Phase: Completed Total Size: 5Gi ActionSet Name: volumesnapshot-for-apecloud-mysql Duration: 68s Start Time: Jun 19,2025 19:36 UTC+0800 Completion Time: Jun 19,2025 19:37 UTC+0800 Time Range Start: Jun 19,2025 19:36 UTC+0800 Time Range End: Jun 19,2025 19:36 UTC+0800 Warning Events: `kbcli cluster restore apemysql-erdpec-backup --backup backup-ns-dmaln-apemysql-erdpec-20250619193635 --namespace ns-dmaln ` Cluster apemysql-erdpec-backup created check cluster status `kbcli cluster list apemysql-erdpec-backup --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec-backup ns-dmaln WipeOut Jun 19,2025 19:37 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec-backup --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-backup-mysql-0 ns-dmaln apemysql-erdpec-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-7-10.us-west-2.compute.internal/172.31.7.10 Jun 19,2025 19:38 UTC+0800 apemysql-erdpec-backup-mysql-1 ns-dmaln apemysql-erdpec-backup mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:38 UTC+0800 apemysql-erdpec-backup-mysql-2 ns-dmaln apemysql-erdpec-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 19:39 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-backup-mysql-1;follower: apemysql-erdpec-backup-mysql-0 apemysql-erdpec-backup-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec-backup --namespace ns-dmaln` check cluster connect done `kbcli cluster describe-backup backup-ns-dmaln-apemysql-erdpec-20250619193635 --namespace ns-dmaln ` Name: backup-ns-dmaln-apemysql-erdpec-20250619193635 Cluster: apemysql-erdpec Namespace: ns-dmaln Spec: Method: volume-snapshot Policy Name: apemysql-erdpec-mysql-backup-policy Status: Phase: Completed Total Size: 5Gi ActionSet Name: volumesnapshot-for-apecloud-mysql Duration: 68s Start Time: Jun 19,2025 19:36 UTC+0800 Completion Time: Jun 19,2025 19:37 UTC+0800 Time Range Start: Jun 19,2025 19:36 UTC+0800 Time Range End: Jun 19,2025 19:36 UTC+0800 Warning Events: cluster connect Connect to instance apemysql-erdpec-backup-mysql-1: out of apemysql-erdpec-backup-mysql-1(leader), apemysql-erdpec-backup-mysql-0(follower), apemysql-erdpec-backup-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-erdpec-backup-mysql-0.apemysql-erdpec-backup-mysql-headless:13306 Follower YES 0 0 2 apemysql-erdpec-backup-mysql-1.apemysql-erdpec-backup-mysql-headless:13306 Leader YES 0 0 3 apemysql-erdpec-backup-mysql-2.apemysql-erdpec-backup-mysql-headless:13306 Follower YES 0 1 `echo 'show variables like "max_connections";' | kbcli cluster connect apemysql-erdpec-backup --namespace ns-dmaln ` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Connect to instance apemysql-erdpec-backup-mysql-1: out of apemysql-erdpec-backup-mysql-1(leader), apemysql-erdpec-backup-mysql-0(follower), apemysql-erdpec-backup-mysql-2(follower) Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster apemysql-erdpec-backup `kbcli cluster delete apemysql-erdpec-backup --auto-approve --namespace ns-dmaln ` Cluster apemysql-erdpec-backup deleted pod_info:apemysql-erdpec-backup-mysql-0 4/4 Running 0 76s apemysql-erdpec-backup-mysql-1 4/4 Running 0 55s apemysql-erdpec-backup-mysql-2 4/4 Running 0 38s pod_info:apemysql-erdpec-backup-mysql-0 3/4 Terminating 0 97s apemysql-erdpec-backup-mysql-1 3/4 Terminating 0 76s apemysql-erdpec-backup-mysql-2 3/4 Terminating 0 59s No resources found in ns-dmaln namespace. delete cluster pod done No resources found in ns-dmaln namespace. check cluster resource non-exist OK: pvc No resources found in ns-dmaln namespace. delete cluster done No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-dmaln-apemysql-erdpec-20250619193635 --namespace ns-dmaln ` backup.dataprotection.kubeblocks.io/backup-ns-dmaln-apemysql-erdpec-20250619193635 patched `kbcli cluster delete-backup apemysql-erdpec --name backup-ns-dmaln-apemysql-erdpec-20250619193635 --force --auto-approve --namespace ns-dmaln ` Backup backup-ns-dmaln-apemysql-erdpec-20250619193635 deleted No opsrequests found in ns-dmaln namespace. cluster xtrabackup backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup apemysql-erdpec --method xtrabackup --namespace ns-dmaln ` Backup backup-ns-dmaln-apemysql-erdpec-20250619194043 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-dmaln-apemysql-erdpec-20250619194043 -n ns-dmaln check backup status `kbcli cluster list-backups apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-dmaln-apemysql-erdpec-20250619194043 ns-dmaln apemysql-erdpec xtrabackup Running Jun 19,2025 19:40 UTC+0800 backup_status:apemysql-erdpec-xtrabackup-Running backup_status:apemysql-erdpec-xtrabackup-Running check backup status done backup_status:backup-ns-dmaln-apemysql-erdpec-20250619194043 ns-dmaln apemysql-erdpec xtrabackup Completed 5899345 10s Jun 19,2025 19:40 UTC+0800 Jun 19,2025 19:40 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "apemysql-erdpec-backup" not found `kbcli cluster describe-backup backup-ns-dmaln-apemysql-erdpec-20250619194043 --namespace ns-dmaln ` Name: backup-ns-dmaln-apemysql-erdpec-20250619194043 Cluster: apemysql-erdpec Namespace: ns-dmaln Spec: Method: xtrabackup Policy Name: apemysql-erdpec-mysql-backup-policy Status: Phase: Completed Total Size: 5899345 ActionSet Name: xtrabackup-for-apecloud-mysql Repository: backuprepo-kbcli-test Duration: 10s Start Time: Jun 19,2025 19:40 UTC+0800 Completion Time: Jun 19,2025 19:40 UTC+0800 Path: /ns-dmaln/apemysql-erdpec-b80a8d6b-8020-4dd3-9818-a27aaf637d9e/mysql/backup-ns-dmaln-apemysql-erdpec-20250619194043 Time Range Start: Jun 19,2025 19:40 UTC+0800 Time Range End: Jun 19,2025 19:40 UTC+0800 Warning Events: `kbcli cluster restore apemysql-erdpec-backup --backup backup-ns-dmaln-apemysql-erdpec-20250619194043 --namespace ns-dmaln ` Cluster apemysql-erdpec-backup created check cluster status `kbcli cluster list apemysql-erdpec-backup --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec-backup ns-dmaln WipeOut Jun 19,2025 19:40 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec-backup --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-backup-mysql-0 ns-dmaln apemysql-erdpec-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 19:41 UTC+0800 apemysql-erdpec-backup-mysql-1 ns-dmaln apemysql-erdpec-backup mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 apemysql-erdpec-backup-mysql-2 ns-dmaln apemysql-erdpec-backup mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:42 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-backup-mysql-1;follower: apemysql-erdpec-backup-mysql-0 apemysql-erdpec-backup-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec-backup --namespace ns-dmaln` check cluster connect done `kbcli cluster describe-backup backup-ns-dmaln-apemysql-erdpec-20250619194043 --namespace ns-dmaln ` Name: backup-ns-dmaln-apemysql-erdpec-20250619194043 Cluster: apemysql-erdpec Namespace: ns-dmaln Spec: Method: xtrabackup Policy Name: apemysql-erdpec-mysql-backup-policy Status: Phase: Completed Total Size: 5899345 ActionSet Name: xtrabackup-for-apecloud-mysql Repository: backuprepo-kbcli-test Duration: 10s Start Time: Jun 19,2025 19:40 UTC+0800 Completion Time: Jun 19,2025 19:40 UTC+0800 Path: /ns-dmaln/apemysql-erdpec-b80a8d6b-8020-4dd3-9818-a27aaf637d9e/mysql/backup-ns-dmaln-apemysql-erdpec-20250619194043 Time Range Start: Jun 19,2025 19:40 UTC+0800 Time Range End: Jun 19,2025 19:40 UTC+0800 Warning Events: cluster connect Connect to instance apemysql-erdpec-backup-mysql-1: out of apemysql-erdpec-backup-mysql-1(leader), apemysql-erdpec-backup-mysql-0(follower), apemysql-erdpec-backup-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. SERVER_ID IP_PORT ROLE CONNECTED LOG_DELAY_NUM APPLY_DELAY_NUM 1 apemysql-erdpec-backup-mysql-0.apemysql-erdpec-backup-mysql-headless:13306 Follower YES 0 0 2 apemysql-erdpec-backup-mysql-1.apemysql-erdpec-backup-mysql-headless:13306 Leader YES 0 0 3 apemysql-erdpec-backup-mysql-2.apemysql-erdpec-backup-mysql-headless:13306 Follower YES 0 0 `echo 'show variables like "max_connections";' | kbcli cluster connect apemysql-erdpec-backup --namespace ns-dmaln ` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. Connect to instance apemysql-erdpec-backup-mysql-1: out of apemysql-erdpec-backup-mysql-1(leader), apemysql-erdpec-backup-mysql-0(follower), apemysql-erdpec-backup-mysql-2(follower) Variable_name Value max_connections 100 connect cluster Success set max_connections to 100 delete cluster apemysql-erdpec-backup `kbcli cluster delete apemysql-erdpec-backup --auto-approve --namespace ns-dmaln ` Cluster apemysql-erdpec-backup deleted pod_info:apemysql-erdpec-backup-mysql-0 4/4 Running 0 86s apemysql-erdpec-backup-mysql-1 4/4 Running 0 62s apemysql-erdpec-backup-mysql-2 4/4 Running 0 48s pod_info:apemysql-erdpec-backup-mysql-0 3/4 Terminating 0 107s apemysql-erdpec-backup-mysql-1 3/4 Terminating 0 83s apemysql-erdpec-backup-mysql-2 3/4 Terminating 0 69s No resources found in ns-dmaln namespace. delete cluster pod done No resources found in ns-dmaln namespace. check cluster resource non-exist OK: pvc No resources found in ns-dmaln namespace. delete cluster done No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. cluster rebulid instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: apemysql-erdpec-rebuildinstance- namespace: ns-dmaln spec: type: RebuildInstance clusterName: apemysql-erdpec force: true rebuildFrom: - componentName: mysql instances: - name: apemysql-erdpec-mysql-0 backupName: backup-ns-dmaln-apemysql-erdpec-20250619194043 inPlace: false restoreEnv: - name: SIGNAL_FILE value: .xtrabackup_rebuild check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_apemysql-erdpec.yaml` opsrequest.apps.kubeblocks.io/apemysql-erdpec-rebuildinstance-tjg4k created create test_ops_cluster_apemysql-erdpec.yaml Success `rm -rf test_ops_cluster_apemysql-erdpec.yaml` check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` No opsrequests found in ns-dmaln namespace. No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: No opsrequests found in ns-dmaln namespace. ops_status: not found ops rebuildinstance check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-dmaln-apemysql-erdpec-20250619194043 --namespace ns-dmaln ` backup.dataprotection.kubeblocks.io/backup-ns-dmaln-apemysql-erdpec-20250619194043 patched `kbcli cluster delete-backup apemysql-erdpec --name backup-ns-dmaln-apemysql-erdpec-20250619194043 --force --auto-approve --namespace ns-dmaln ` Backup backup-ns-dmaln-apemysql-erdpec-20250619194043 deleted No opsrequests found in ns-dmaln namespace. cluster list-logs `kbcli cluster list-logs apemysql-erdpec --namespace ns-dmaln ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update apemysql-erdpec --enable-all-logs=true --namespace ns-dmaln Error from server (NotFound): pods "apemysql-erdpec-mysql-1" not found cluster logs `kbcli cluster logs apemysql-erdpec --tail 30 --namespace ns-dmaln ` Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_MEMBER_HOST=apemysql-erdpec-mysql-1.apemysql-erdpec-mysql-headless 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_LEADER_HOST= 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_DATA_DIR=/data/mysql/data 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_LOG_DIR=/data/mysql/log 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_EXTRA_CONF_FILE=/opt/mysql/my.cnf 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_PORT=3306 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_CONSENSUS_PORT=13306 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: MYSQL_CLUSTER_INFO=apemysql-erdpec-mysql-0.apemysql-erdpec-mysql-headless:13306;apemysql-erdpec-mysql-1.apemysql-erdpec-mysql-headless:13306;apemysql-erdpec-mysql-2.apemysql-erdpec-mysql-headless:13306@2 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: Injecting configuration /opt/mysql/my.cnf to /etc/my.cnf 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: Injecting configuration /etc/mysql/wesql_cluster.cnf to /etc/my.cnf 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: DATADIR=/data/mysql/data/ 2025-06-19 11:24:51+00:00 [Note] [Apecloud MySQL]: PID_FILE=/var/run/mysqld/mysqld.pid 2025-06-19 11:24:52+00:00 [Note] [Apecloud MySQL]: SOCKET=/var/run/mysqld/mysqld.sock 2025-06-19 11:24:53+00:00 [Note] [Apecloud MySQL]: Generating post logrotate shell script: /apecloud/mysql/scripts//post_logroate.sh 2025-06-19 11:24:53+00:00 [Note] [Apecloud MySQL]: error_log=/data/mysql/log/mysqld-error.log 2025-06-19 11:24:53+00:00 [Note] [Apecloud MySQL]: slow_log=/data/mysql/data/apemysql-erdpec-mysql-1-slow.log 2025-06-19 11:24:53+00:00 [Note] [Apecloud MySQL]: general_log=/data/mysql/data/apemysql-erdpec-mysql-1.log 2025-06-19 11:24:54+00:00 [Note] [Apecloud MySQL]: Switching to dedicated user 'mysql' 2025-06-19 11:24:54+00:00 [Note] [MySQL]: Entrypoint script for MySQL Server started. 2025-06-19 11:24:54+00:00 [Note] [MySQL]: KB_EMBEDDED_WESQL=1 2025-06-19 11:24:54+00:00 [Note] [Apecloud MySQL]: DATADIR=/data/mysql/data/ 2025-06-19 11:24:54+00:00 [Note] [Apecloud MySQL]: PID_FILE=/var/run/mysqld/mysqld.pid 2025-06-19 11:24:55+00:00 [Note] [Apecloud MySQL]: SOCKET=/var/run/mysqld/mysqld.sock 2025-06-19 11:24:56+00:00 [Note] [Apecloud MySQL]: JEMALLOC_ENABLED=1 2025-06-19 11:24:56+00:00 [Note] [Apecloud MySQL]: Using jemalloc: LD_PRELOAD=/usr/local/lib/libjemalloc.so.2 2025-06-19 11:24:56+00:00 [Note] [Apecloud MySQL]: Using persisted data '/var/lib/mysql/mysql.sock' -> '/var/run/mysqld/mysqld.sock' 2025-06-19 11:24:56+00:00 [Note] [Apecloud MySQL]: Starting WeSQL server: 'mysqld' cluster logs slow `kbcli cluster logs apemysql-erdpec --tail 30 --file-type=slow --namespace ns-dmaln ` error: get pod component definition name in cluster.yaml fail cluster logs error `kbcli cluster logs apemysql-erdpec --tail 30 --file-type=error --namespace ns-dmaln ` error: get pod component definition name in cluster.yaml fail cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose apemysql-erdpec --auto-approve --force=true --type internet --enable false --components mysql --namespace ns-dmaln ` OpsRequest apemysql-erdpec-expose-znc2t created successfully, you can view the progress: kbcli cluster describe-ops apemysql-erdpec-expose-znc2t -n ns-dmaln check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-expose-znc2t ns-dmaln Expose apemysql-erdpec mysql Running 0/1 Jun 19,2025 19:48 UTC+0800 check cluster status `kbcli cluster list apemysql-erdpec --show-labels --namespace ns-dmaln ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS apemysql-erdpec ns-dmaln WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=apemysql-erdpec check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances apemysql-erdpec --namespace ns-dmaln ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME apemysql-erdpec-mysql-0 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-14-205.us-west-2.compute.internal/172.31.14.205 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-1 ns-dmaln apemysql-erdpec mysql Running leader us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 19:24 UTC+0800 apemysql-erdpec-mysql-2 ns-dmaln apemysql-erdpec mysql Running follower us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-5-129.us-west-2.compute.internal/172.31.5.129 Jun 19,2025 19:29 UTC+0800 check pod status done check cluster role check cluster role done leader: apemysql-erdpec-mysql-1;follower: apemysql-erdpec-mysql-0 apemysql-erdpec-mysql-2 check cluster connect `echo '' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln` check cluster connect done check ops status `kbcli cluster list-ops apemysql-erdpec --status all --namespace ns-dmaln ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME apemysql-erdpec-expose-znc2t ns-dmaln Expose apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 19:48 UTC+0800 check ops status done ops_status:apemysql-erdpec-expose-znc2t ns-dmaln Expose apemysql-erdpec mysql Succeed 1/1 Jun 19,2025 19:48 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests apemysql-erdpec-expose-znc2t --namespace ns-dmaln ` opsrequest.apps.kubeblocks.io/apemysql-erdpec-expose-znc2t patched `kbcli cluster delete-ops --name apemysql-erdpec-expose-znc2t --force --auto-approve --namespace ns-dmaln ` OpsRequest apemysql-erdpec-expose-znc2t deleted `echo 'use mydb; SELECT value FROM tmp_table WHERE id = 1;' | kbcli cluster connect apemysql-erdpec --namespace ns-dmaln ` check data: Connect to instance apemysql-erdpec-mysql-1: out of apemysql-erdpec-mysql-1(leader), apemysql-erdpec-mysql-0(follower), apemysql-erdpec-mysql-2(follower) Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql `echo "echo \"use mydb; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5'" | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly data: Defaulted container "mysql" out of: mysql, mysql-exporter, lorry, config-manager, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: arning] Using a password on the command line interface can be insecure. value iuktg check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -hapemysql-erdpec-mysql.ns-dmaln.svc.cluster.local -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-1 --namespace ns-dmaln -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=apemysql-erdpec` set secret: apemysql-erdpec-mysql-account-root `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.username***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.password***"` `kubectl get secrets apemysql-erdpec-mysql-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:TR93uUN6pA3234g5;DB_PORT:3306;DB_DATABASE:mysql check readonly db_client batch data count `echo "echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -P3306 -h127.0.0.1 -uroot -p'TR93uUN6pA3234g5' " | kubectl exec -it apemysql-erdpec-mysql-0 --namespace ns-dmaln -- bash ` check readonly db_client batch data Success delete cluster apemysql-erdpec `kbcli cluster delete apemysql-erdpec --auto-approve --namespace ns-dmaln ` Cluster apemysql-erdpec deleted pod_info:apemysql-erdpec-mysql-0 4/4 Running 0 24m apemysql-erdpec-mysql-1 4/4 Running 0 24m apemysql-erdpec-mysql-2 4/4 Running 0 19m pod_info:apemysql-erdpec-mysql-0 3/4 Terminating 0 24m apemysql-erdpec-mysql-1 3/4 Terminating 0 24m apemysql-erdpec-mysql-2 3/4 Terminating 0 19m No resources found in ns-dmaln namespace. delete cluster pod done No resources found in ns-dmaln namespace. check cluster resource non-exist OK: pvc No resources found in ns-dmaln namespace. delete cluster done No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. No resources found in ns-dmaln namespace. ApeCloud Mysql Test Suite All Done! --------------------------------------ApeCloud Mysql (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=apecloud-mysql;ClusterVersion=ac-mysql-8.0.30-1;]|[Description=Create a cluster with the specified cluster definition apecloud-mysql and cluster version ac-mysql-8.0.30-1] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=iuktg]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=iuktg;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Expose]|[Enable=true;TYPE=internet;ComponentName=mysql]|[Description=Expose Enable the internet service with mysql component] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=mysql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=mysql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[HorizontalScaling Out]|[ComponentName=mysql]|[Description=HorizontalScaling Out the cluster specify component mysql] [PASSED]|[HorizontalScaling In]|[ComponentName=mysql]|[Description=HorizontalScaling In the cluster specify component mysql] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=mysql]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mysql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=Full CPU Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=mysql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[VerticalScaling]|[ComponentName=mysql]|[Description=VerticalScaling the cluster specify component mysql] [PASSED]|[Reconfiguring]|[ComponentName=mysql;consensus_log_level=LOG_INFO,log_error_verbosity=3]|[Description=Reconfiguring the cluster specify component mysql set consensus_log_level=LOG_INFO,log_error_verbosity=3] [FAILED]|[SwitchOver]|[ComponentName=mysql]|[Description=SwitchOver the cluster specify component mysql] [PASSED]|[Bench]|[ComponentName=mysql]|[Description=Bench the cluster service with mysql component] [PASSED]|[Bench]|[HostType=LB;ComponentName=mysql]|[Description=Bench the cluster LB service with mysql component] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=mysql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Reconfiguring]|[ComponentName=mysql;max_connections=1000,general_log=OFF]|[Description=Reconfiguring the cluster specify component mysql set max_connections=1000,general_log=OFF] [PASSED]|[Reconfiguring]|[ComponentName=mysql;max_connections=2000,general_log=ON,innodb_sort_buffer_size=2097152]|[Description=Reconfiguring the cluster specify component mysql set max_connections=2000,general_log=ON,innodb_sort_buffer_size=2097152] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=mysql]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[HscaleOfflineInstances]|[ComponentName=mysql]|[Description=Hscale the cluster instances offline specify component mysql] [PASSED]|[HscaleOnlineInstances]|[ComponentName=mysql]|[Description=Hscale the cluster instances online specify component mysql] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Network Delay Failover;Durations=2m;ComponentName=mysql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=mysql]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[VolumeExpansion]|[ComponentName=mysql]|[Description=VolumeExpansion the cluster specify component mysql] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=xtrabackup]|[Description=The cluster Schedule xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=xtrabackup]|[Description=Delete the Schedule xtrabackup restore cluster] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Backup]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Backup] [PASSED]|[Restore]|[BackupMethod=xtrabackup]|[Description=The cluster xtrabackup Restore] [PASSED]|[Connect]|[ComponentName=mysql]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=xtrabackup]|[Description=Delete the xtrabackup restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=mysql]|[Description=Rebuild the cluster instance specify component mysql] [PASSED]|[Expose]|[Disable=true;TYPE=internet;ComponentName=mysql]|[Description=Expose Disable the internet service with mysql component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]