bash test/kbcli/test_kbcli_0.9.sh --type 5 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-rxzwd ` `kubectl create namespace ns-rxzwd` namespace/ns-rxzwd created create namespace ns-rxzwd done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 7 32.1M 7 2332k 0 0 8540k 0 0:00:03 --:--:-- 0:00:03 8540k 100 32.1M 100 32.1M 0 0 80.2M 0 --:--:-- --:--:-- --:--:-- 233M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:redis set component version set component version:redis set service versions:5.0.12,6.2.14,6.2.17,7.0.6,7.2.10,7.2.4,7.2.7,7.4.2,7.4.5,8.0.1,8.0.3 set service versions sorted:5.0.12,6.2.14,6.2.17,7.0.6,7.2.4,7.2.7,7.2.10,7.4.2,7.4.5,8.0.1,8.0.3 set redis component definition set redis component definition redis-7 REPORT_COUNT 0:0 set replicas first:2,5.0.12|2,6.2.14|2,6.2.17|2,7.0.6|2,7.2.4|2,7.2.7|2,7.2.10|2,7.4.2|2,7.4.5|2,8.0.1|2,8.0.3 set replicas third:2,7.0.6 set replicas fourth:2,7.0.6 set minimum cmpv service version set minimum cmpv service version replicas:2,7.0.6 REPORT_COUNT:1 CLUSTER_TOPOLOGY:replication topology replication found in cluster definition redis LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-rxzwd namespace. termination_policy:DoNotTerminate create 2 replica DoNotTerminate redis cluster check cluster version check cluster definition check component definition set component definition by component version check cmpd by labels set component definition1: redis-7 by component version:redis apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: redis-lyqvhu namespace: ns-rxzwd spec: terminationPolicy: DoNotTerminate componentSpecs: - name: redis componentDef: redis-7 replicas: 2 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi switchPolicy: type: Noop volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: redis-sentinel componentDef: redis-sentinel-7 replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: redis-twemproxy componentDef: redis-twemproxy-0.5 replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_redis-lyqvhu.yaml` cluster.apps.kubeblocks.io/redis-lyqvhu created apply test_create_redis-lyqvhu.yaml Success `rm -rf test_create_redis-lyqvhu.yaml` check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate ConditionsError Sep 01,2025 11:24 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check pod redis-lyqvhu-redis-0 container_name redis exist password wVrHR32762 check pod redis-lyqvhu-redis-0 container_name lorry exist password wVrHR32762 check pod redis-lyqvhu-redis-0 container_name config-manager exist password wVrHR32762 No container logs contain secret password. describe cluster `kbcli cluster describe redis-lyqvhu --namespace ns-rxzwd ` Name: redis-lyqvhu Created Time: Sep 01,2025 11:24 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-rxzwd Running DoNotTerminate Endpoints: COMPONENT MODE INTERNAL EXTERNAL redis ReadWrite redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local:6379 redis-sentinel ReadWrite redis-lyqvhu-redis-sentinel-redis-sentinel.ns-rxzwd.svc.cluster.local:26379 redis-twemproxy ReadWrite redis-lyqvhu-redis-twemproxy-twemproxy.ns-rxzwd.svc.cluster.local:22121 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME redis redis-lyqvhu-redis-0 primary Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis redis-lyqvhu-redis-1 secondary Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-sentinel redis-lyqvhu-redis-sentinel-0 Running 0 aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-sentinel redis-lyqvhu-redis-sentinel-1 Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-sentinel redis-lyqvhu-redis-sentinel-2 Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-twemproxy redis-lyqvhu-redis-twemproxy-0 Running 0 aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-twemproxy redis-lyqvhu-redis-twemproxy-1 Running 0 aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-twemproxy redis-lyqvhu-redis-twemproxy-2 Running 0 aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS redis false 100m / 100m 512Mi / 512Mi data:1Gi default redis-sentinel false 100m / 100m 512Mi / 512Mi data:1Gi default redis-twemproxy false 100m / 100m 512Mi / 512Mi data:1Gi default Images: COMPONENT TYPE IMAGE redis docker.io/apecloud/redis-stack-server:7.4.0-v6 redis-sentinel docker.io/apecloud/redis-stack-server:7.4.0-v6 redis-twemproxy docker.io/apecloud/twemproxy:0.5.0 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-rxzwd redis-lyqvhu `kbcli cluster label redis-lyqvhu app.kubernetes.io/instance- --namespace ns-rxzwd ` label "app.kubernetes.io/instance" not found. `kbcli cluster label redis-lyqvhu app.kubernetes.io/instance=redis-lyqvhu --namespace ns-rxzwd ` `kbcli cluster label redis-lyqvhu --list --namespace ns-rxzwd ` NAME NAMESPACE LABELS redis-lyqvhu ns-rxzwd app.kubernetes.io/instance=redis-lyqvhu label cluster app.kubernetes.io/instance=redis-lyqvhu Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=redis-lyqvhu --namespace ns-rxzwd ` `kbcli cluster label redis-lyqvhu --list --namespace ns-rxzwd ` NAME NAMESPACE LABELS redis-lyqvhu ns-rxzwd app.kubernetes.io/instance=redis-lyqvhu case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label redis-lyqvhu case.name=kbcli.test2 --overwrite --namespace ns-rxzwd ` `kbcli cluster label redis-lyqvhu --list --namespace ns-rxzwd ` NAME NAMESPACE LABELS redis-lyqvhu ns-rxzwd app.kubernetes.io/instance=redis-lyqvhu case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label redis-lyqvhu case.name- --namespace ns-rxzwd ` `kbcli cluster label redis-lyqvhu --list --namespace ns-rxzwd ` NAME NAMESPACE LABELS redis-lyqvhu ns-rxzwd app.kubernetes.io/instance=redis-lyqvhu delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.4.5 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:b6b2e66014f8e765 redis_mode:standalone os:Linux 5.15.0-1092-azure x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:33b052b3620b82d721a47a7aab7167786374b8fe tcp_port:6379 server_time_usec:1756697298146195 uptime_in_seconds:97 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:11866834 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo 'echo "CONFIG GET maxclients" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-redis-lyqvhu" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-redis-lyqvhu --namespace ns-rxzwd ` Error from server (NotFound): pods "test-db-client-executionloop-redis-lyqvhu" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-redis-lyqvhu" not found `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-redis-lyqvhu namespace: ns-rxzwd spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local" - "--user" - "default" - "--password" - "wVrHR32762" - "--port" - "6379" - "--dbtype" - "redis" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-redis-lyqvhu.yaml` pod/test-db-client-executionloop-redis-lyqvhu created apply test-db-client-executionloop-redis-lyqvhu.yaml Success `rm -rf test-db-client-executionloop-redis-lyqvhu.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 21s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 27s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 32s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 38s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 43s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 49s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 54s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 1/1 Running 0 60s check pod test-db-client-executionloop-redis-lyqvhu status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-lyqvhu 0/1 Completed 0 65s check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done Execution loop start: Execution loop start:set executions_loop_key executions_loop_test_1 [ 1s ] executions total: 371 successful: 371 failed: 0 disconnect: 0 [ 2s ] executions total: 794 successful: 794 failed: 0 disconnect: 0 [2025-09-01 03:28:35.098] Connection error occurred! [ 3s ] executions total: 801 successful: 800 failed: 1 disconnect: 1 [2025-09-01 03:28:35.098] Connection error occurred! [2025-09-01 03:28:36.102] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 4s ] executions total: 802 successful: 801 failed: 1 disconnect: 1 [ 5s ] executions total: 1263 successful: 1262 failed: 1 disconnect: 1 [2025-09-01 03:28:40.592] Connection error occurred! [ 6s ] executions total: 1602 successful: 1600 failed: 2 disconnect: 2 [2025-09-01 03:28:40.592] Connection error occurred! [2025-09-01 03:28:41.637] Connection successfully recovered! The connection was restored in 1045 milliseconds. [ 7s ] executions total: 1603 successful: 1601 failed: 2 disconnect: 2 [ 8s ] executions total: 2085 successful: 2083 failed: 2 disconnect: 2 [2025-09-01 03:28:46.180] Connection error occurred! [ 9s ] executions total: 2403 successful: 2400 failed: 3 disconnect: 3 [2025-09-01 03:28:46.180] Connection error occurred! [2025-09-01 03:28:47.183] Connection successfully recovered! The connection was restored in 1003 milliseconds. [ 10s ] executions total: 2404 successful: 2401 failed: 3 disconnect: 3 [ 11s ] executions total: 3072 successful: 3069 failed: 3 disconnect: 3 [2025-09-01 03:28:51.416] Connection error occurred! [ 12s ] executions total: 3204 successful: 3200 failed: 4 disconnect: 4 [2025-09-01 03:28:51.416] Connection error occurred! [2025-09-01 03:28:52.433] Connection successfully recovered! The connection was restored in 1017 milliseconds. [ 13s ] executions total: 3205 successful: 3201 failed: 4 disconnect: 4 [ 14s ] executions total: 3712 successful: 3708 failed: 4 disconnect: 4 [2025-09-01 03:28:56.977] Connection error occurred! [ 15s ] executions total: 4005 successful: 4000 failed: 5 disconnect: 5 [2025-09-01 03:28:56.977] Connection error occurred! [2025-09-01 03:28:57.981] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 16s ] executions total: 4006 successful: 4001 failed: 5 disconnect: 5 [ 17s ] executions total: 4523 successful: 4518 failed: 5 disconnect: 5 [2025-09-01 03:29:02.493] Connection error occurred! [ 18s ] executions total: 4806 successful: 4800 failed: 6 disconnect: 6 [2025-09-01 03:29:02.493] Connection error occurred! [2025-09-01 03:29:03.501] Connection successfully recovered! The connection was restored in 1008 milliseconds. [ 19s ] executions total: 4807 successful: 4801 failed: 6 disconnect: 6 [ 20s ] executions total: 5341 successful: 5335 failed: 6 disconnect: 6 [2025-09-01 03:29:07.999] Connection error occurred! [ 21s ] executions total: 5607 successful: 5600 failed: 7 disconnect: 7 [2025-09-01 03:29:07.999] Connection error occurred! [2025-09-01 03:29:09.002] Connection successfully recovered! The connection was restored in 1003 milliseconds. [ 22s ] executions total: 5608 successful: 5601 failed: 7 disconnect: 7 [ 23s ] executions total: 6178 successful: 6171 failed: 7 disconnect: 7 [2025-09-01 03:29:13.413] Connection error occurred! [ 24s ] executions total: 6408 successful: 6400 failed: 8 disconnect: 8 [2025-09-01 03:29:13.413] Connection error occurred! [2025-09-01 03:29:14.417] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 25s ] executions total: 6409 successful: 6401 failed: 8 disconnect: 8 [ 26s ] executions total: 6967 successful: 6959 failed: 8 disconnect: 8 [2025-09-01 03:29:18.830] Connection error occurred! [ 27s ] executions total: 7209 successful: 7200 failed: 9 disconnect: 9 [2025-09-01 03:29:18.830] Connection error occurred! [2025-09-01 03:29:19.832] Connection successfully recovered! The connection was restored in 1002 milliseconds. [ 28s ] executions total: 7210 successful: 7201 failed: 9 disconnect: 9 [ 29s ] executions total: 7855 successful: 7846 failed: 9 disconnect: 9 [2025-09-01 03:29:24.084] Connection error occurred! [ 30s ] executions total: 8010 successful: 8000 failed: 10 disconnect: 10 [2025-09-01 03:29:24.084] Connection error occurred! [2025-09-01 03:29:25.091] Connection successfully recovered! The connection was restored in 1007 milliseconds. [ 31s ] executions total: 8011 successful: 8001 failed: 10 disconnect: 10 [ 32s ] executions total: 8444 successful: 8434 failed: 10 disconnect: 10 [2025-09-01 03:29:29.914] Connection error occurred! [ 33s ] executions total: 8811 successful: 8800 failed: 11 disconnect: 11 [2025-09-01 03:29:29.914] Connection error occurred! [2025-09-01 03:29:30.918] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 60s ] executions total: 8812 successful: 8801 failed: 11 disconnect: 11 Test Result: Total Executions: 8812 Successful Executions: 8801 Failed Executions: 11 Disconnection Counts: 11 Connection Information: Database Type: redis Host: redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local Port: 6379 Database: Table: User: default Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds Key: DB_CLIENT_BATCH_DATA_COUNT: 8801 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-redis-lyqvhu --namespace ns-rxzwd ` pod/test-db-client-executionloop-redis-lyqvhu patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-redis-lyqvhu" force deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo 'echo "set mykey \"fuqur\"" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. OK add consistent data fuqur Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo 'echo "set mykey \"fuqur\"" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. READONLY You can't write against a read only replica. check add consistent data readonly Success cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose redis-lyqvhu --auto-approve --force=true --type vpc --enable true --components redis --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-expose-czhhp created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-expose-czhhp -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 11:30 UTC+0800 check ops status done ops_status:redis-lyqvhu-expose-czhhp ns-rxzwd Expose redis-lyqvhu redis Succeed 1/1 Sep 01,2025 11:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-expose-czhhp --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-expose-czhhp patched `kbcli cluster delete-ops --name redis-lyqvhu-expose-czhhp --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-expose-czhhp deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-lyqvhu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 action: partition mode: all target: mode: all selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-redis-lyqvhu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-redis-lyqvhu created apply test-chaos-mesh-networkpartition-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-networkpartition-redis-lyqvhu.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-lyqvhu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-0 failover networkpartition Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover drainnode check cluster status before cluster-failover-drainnode check cluster status done cluster_status:Running check node drain check node drain success kubectl get pod redis-lyqvhu-redis-0 --namespace ns-rxzwd -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-38368475-vmss000000 success check if multiple pods are on the same node kubectl get pod redis-lyqvhu-redis-1 --namespace ns-rxzwd -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-38368475-vmss000000 success Multiple pods on the same node cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale redis-lyqvhu --auto-approve --force=true --components redis --cpu 200m --memory 0.6Gi --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-verticalscaling-wkfj8 created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-verticalscaling-wkfj8 -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-verticalscaling-wkfj8 ns-rxzwd VerticalScaling redis-lyqvhu redis Running -/- Sep 01,2025 11:34 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:35 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:34 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-verticalscaling-wkfj8 ns-rxzwd VerticalScaling redis-lyqvhu redis Succeed 2/2 Sep 01,2025 11:34 UTC+0800 check ops status done ops_status:redis-lyqvhu-verticalscaling-wkfj8 ns-rxzwd VerticalScaling redis-lyqvhu redis Succeed 2/2 Sep 01,2025 11:34 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-verticalscaling-wkfj8 --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-verticalscaling-wkfj8 patched `kbcli cluster delete-ops --name redis-lyqvhu-verticalscaling-wkfj8 --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-verticalscaling-wkfj8 deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success 9 cluster configure component_tmp: redis check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure redis-lyqvhu --auto-approve --force=true --set maxclients=10001 --components redis --config-spec redis-replication-config --config-file redis.conf --namespace ns-rxzwd ` Will updated configure file meta: ConfigSpec: redis-replication-config ConfigFile: redis.conf ComponentName: redis ClusterName: redis-lyqvhu OpsRequest redis-lyqvhu-reconfiguring-72r2x created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-reconfiguring-72r2x -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-reconfiguring-72r2x ns-rxzwd Reconfiguring redis-lyqvhu redis Running -/- Sep 01,2025 11:36 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-reconfiguring-72r2x ns-rxzwd Reconfiguring redis-lyqvhu redis Succeed 2/2 Sep 01,2025 11:36 UTC+0800 check ops status done ops_status:redis-lyqvhu-reconfiguring-72r2x ns-rxzwd Reconfiguring redis-lyqvhu redis Succeed 2/2 Sep 01,2025 11:36 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-reconfiguring-72r2x --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-reconfiguring-72r2x patched `kbcli cluster delete-ops --name redis-lyqvhu-reconfiguring-72r2x --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-reconfiguring-72r2x deleted check config variables Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. current value_actual: 10001 configure:[maxclients] result actual:[10001] equal expected:[10001] `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-lyqvhu" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-redis-lyqvhu.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-redis-lyqvhu created apply test-chaos-mesh-podfailure-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-podfailure-redis-lyqvhu.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Abnormal Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:42 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-lyqvhu" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-redis-lyqvhu patched check failover pod name failover pod name:redis-lyqvhu-redis-1 failover podfailure Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success cluster update monitor cluster does not need to check monitor currently check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:42 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-lyqvhu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-redis-lyqvhu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-redis-lyqvhu created apply test-chaos-mesh-networkduplicate-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-networkduplicate-redis-lyqvhu.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-sentinel-1" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-sentinel-2" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:47 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:47 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-lyqvhu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-1 failover networkduplicate Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-lyqvhu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-redis-lyqvhu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-redis-lyqvhu created apply test-chaos-mesh-networkbandwidthover-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-redis-lyqvhu.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:47 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 11:47 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-lyqvhu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-0 failover networkbandwidthover Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-lyqvhu" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-redis-lyqvhu.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-redis-lyqvhu created apply test-chaos-mesh-oom-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-oom-redis-lyqvhu.yaml` check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-0" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-sentinel-1" force deleted Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-sentinel-2" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-lyqvhu" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-1 failover oom Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success cluster update enable-all-logs `kbcli cluster update redis-lyqvhu --enable-all-logs=true --namespace ns-rxzwd ` error: name is required check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-redis-lyqvhu" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-redis-lyqvhu --namespace ns-rxzwd ` Error from server (NotFound): pods "test-db-client-connectionstress-redis-lyqvhu" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-redis-lyqvhu" not found `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-redis-lyqvhu namespace: ns-rxzwd spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local" - "--user" - "default" - "--password" - "wVrHR32762" - "--port" - "6379" - "--database" - "db0" - "--dbtype" - "redis" - "--test" - "connectionstress" - "--connections" - "10000" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-redis-lyqvhu.yaml` pod/test-db-client-connectionstress-redis-lyqvhu created apply test-db-client-connectionstress-redis-lyqvhu.yaml Success `rm -rf test-db-client-connectionstress-redis-lyqvhu.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 0/1 ContainerCreating 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 21s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 26s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 32s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 37s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 43s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 48s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 54s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 59s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 1/1 Running 0 65s check pod test-db-client-connectionstress-redis-lyqvhu status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-lyqvhu 0/1 Completed 0 70s check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000007/10.224.0.9 Sep 01,2025 11:44 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done --host redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local --user default --password wVrHR32762 --port 6379 --database db0 --dbtype redis --test connectionstress --connections 10000 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Test Result: Connection stress test results: Duration: 60 seconds Successful connections: 10058 Failed connections: 0 Connection Information: Database Type: redis Host: redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local Port: 6379 Database: db0 Table: User: default Org: Access Mode: mysql Test Type: connectionstress Connection Count: 10000 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-redis-lyqvhu --namespace ns-rxzwd ` pod/test-db-client-connectionstress-redis-lyqvhu patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-redis-lyqvhu" force deleted check failover pod name failover pod name:redis-lyqvhu-redis-1 failover connectionstress Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-lyqvhu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-redis-lyqvhu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-redis-lyqvhu created apply test-chaos-mesh-networklossover-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-networklossover-redis-lyqvhu.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-lyqvhu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-0 failover networklossover Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-lyqvhu" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-redis-lyqvhu.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-redis-lyqvhu created apply test-chaos-mesh-timeoffset-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-timeoffset-redis-lyqvhu.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:37 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 11:24 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:57 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:01 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:25 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-lyqvhu" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-0 failover timeoffset Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop redis-lyqvhu --auto-approve --force=true --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-stop-h7mrc created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-stop-h7mrc -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-stop-h7mrc ns-rxzwd Stop redis-lyqvhu redis,redis-sentinel,redis-twemproxy Running 0/8 Sep 01,2025 12:05 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Stopping Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-stop-h7mrc ns-rxzwd Stop redis-lyqvhu redis,redis-sentinel,redis-twemproxy Succeed 8/8 Sep 01,2025 12:05 UTC+0800 check ops status done ops_status:redis-lyqvhu-stop-h7mrc ns-rxzwd Stop redis-lyqvhu redis,redis-sentinel,redis-twemproxy Succeed 8/8 Sep 01,2025 12:05 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-stop-h7mrc --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-stop-h7mrc patched `kbcli cluster delete-ops --name redis-lyqvhu-stop-h7mrc --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-stop-h7mrc deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start redis-lyqvhu --force=true --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-start-c4qmj created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-start-c4qmj -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-start-c4qmj ns-rxzwd Start redis-lyqvhu redis,redis-sentinel,redis-twemproxy Running 0/8 Sep 01,2025 12:07 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:07 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:07 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:07 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:07 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:07 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:07 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:08 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:08 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-start-c4qmj ns-rxzwd Start redis-lyqvhu redis,redis-sentinel,redis-twemproxy Succeed 8/8 Sep 01,2025 12:07 UTC+0800 check ops status done ops_status:redis-lyqvhu-start-c4qmj ns-rxzwd Start redis-lyqvhu redis,redis-sentinel,redis-twemproxy Succeed 8/8 Sep 01,2025 12:07 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-start-c4qmj --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-start-c4qmj patched `kbcli cluster delete-ops --name redis-lyqvhu-start-c4qmj --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-start-c4qmj deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart redis-lyqvhu --auto-approve --force=true --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-restart-msqcm created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-restart-msqcm -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-restart-msqcm ns-rxzwd Restart redis-lyqvhu redis,redis-sentinel,redis-twemproxy Running 0/8 Sep 01,2025 12:09 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-restart-msqcm ns-rxzwd Restart redis-lyqvhu redis,redis-sentinel,redis-twemproxy Succeed 8/8 Sep 01,2025 12:09 UTC+0800 check ops status done ops_status:redis-lyqvhu-restart-msqcm ns-rxzwd Restart redis-lyqvhu redis,redis-sentinel,redis-twemproxy Succeed 8/8 Sep 01,2025 12:09 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-restart-msqcm --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-restart-msqcm patched `kbcli cluster delete-ops --name redis-lyqvhu-restart-msqcm --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-restart-msqcm deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-lyqvhu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-redis-lyqvhu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-redis-lyqvhu created apply test-chaos-mesh-networkdelay-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-networkdelay-redis-lyqvhu.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-lyqvhu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-1 failover networkdelay Failure `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running check node drain check node drain success `kill 1` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-lyqvhu-redis-1" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done check failover pod name failover pod name:redis-lyqvhu-redis-0 failover kill1 Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-lyqvhu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-redis-lyqvhu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-redis-lyqvhu created apply test-chaos-mesh-networkcorruptover-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-redis-lyqvhu.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:09 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-lyqvhu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-1 failover networkcorruptover Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=redis-lyqvhu,apps.kubeblocks.io/component-name=redis,apps.kubeblocks.io/vct-name=data --namespace ns-rxzwd ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in redis-lyqvhu namespace. `kbcli cluster volume-expand redis-lyqvhu --auto-approve --force=true --components redis --volume-claim-templates data --storage 4Gi --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-volumeexpansion-8wn9v created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-volumeexpansion-8wn9v -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-volumeexpansion-8wn9v ns-rxzwd VolumeExpansion redis-lyqvhu redis Running 0/2 Sep 01,2025 12:21 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done No resources found in redis-lyqvhu namespace. check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-volumeexpansion-8wn9v ns-rxzwd VolumeExpansion redis-lyqvhu redis Succeed 2/2 Sep 01,2025 12:21 UTC+0800 check ops status done ops_status:redis-lyqvhu-volumeexpansion-8wn9v ns-rxzwd VolumeExpansion redis-lyqvhu redis Succeed 2/2 Sep 01,2025 12:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-volumeexpansion-8wn9v --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-volumeexpansion-8wn9v patched `kbcli cluster delete-ops --name redis-lyqvhu-volumeexpansion-8wn9v --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-volumeexpansion-8wn9v deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-lyqvhu" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-redis-lyqvhu.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-redis-lyqvhu created apply test-chaos-mesh-dnsrandom-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-dnsrandom-redis-lyqvhu.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-lyqvhu" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-1 failover dnsrandom Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-lyqvhu" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-redis-lyqvhu.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-redis-lyqvhu created apply test-chaos-mesh-dnserror-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-dnserror-redis-lyqvhu.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-lyqvhu" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-1 failover dnserror Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in redis-lyqvhu namespace. `kbcli cluster hscale redis-lyqvhu --auto-approve --force=true --components redis --replicas 3 --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-horizontalscaling-brrv4 created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-horizontalscaling-brrv4 -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-horizontalscaling-brrv4 ns-rxzwd HorizontalScaling redis-lyqvhu redis Running 0/1 Sep 01,2025 12:38 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-2 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:38 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 redis-lyqvhu-redis-2 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done No resources found in redis-lyqvhu namespace. check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-horizontalscaling-brrv4 ns-rxzwd HorizontalScaling redis-lyqvhu redis Succeed 1/1 Sep 01,2025 12:38 UTC+0800 check ops status done ops_status:redis-lyqvhu-horizontalscaling-brrv4 ns-rxzwd HorizontalScaling redis-lyqvhu redis Succeed 1/1 Sep 01,2025 12:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-horizontalscaling-brrv4 --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-horizontalscaling-brrv4 patched `kbcli cluster delete-ops --name redis-lyqvhu-horizontalscaling-brrv4 --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-horizontalscaling-brrv4 deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary), redis-lyqvhu-redis-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success get redis-sentinel replicas get redis-twemproxy replicas cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in redis-lyqvhu namespace. `kbcli cluster hscale redis-lyqvhu --auto-approve --force=true --components redis --replicas 2 --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-horizontalscaling-qt5jh created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-horizontalscaling-qt5jh -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-horizontalscaling-qt5jh ns-rxzwd HorizontalScaling redis-lyqvhu redis Running 0/1 Sep 01,2025 12:40 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done No resources found in redis-lyqvhu namespace. check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-horizontalscaling-qt5jh ns-rxzwd HorizontalScaling redis-lyqvhu redis Succeed 1/1 Sep 01,2025 12:40 UTC+0800 check ops status done ops_status:redis-lyqvhu-horizontalscaling-qt5jh ns-rxzwd HorizontalScaling redis-lyqvhu redis Succeed 1/1 Sep 01,2025 12:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-horizontalscaling-qt5jh --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-horizontalscaling-qt5jh patched `kbcli cluster delete-ops --name redis-lyqvhu-horizontalscaling-qt5jh --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-horizontalscaling-qt5jh deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success test failover fullcpuover check cluster status before cluster-failover-fullcpuover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-lyqvhu" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-lyqvhu.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-lyqvhu created apply test-chaos-mesh-fullcpuover-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-lyqvhu.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-0;secondary: redis-lyqvhu-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-lyqvhu" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-lyqvhu" not found check failover pod name failover pod name:redis-lyqvhu-redis-0 failover fullcpuover Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-0: out of redis-lyqvhu-redis-0(primary), redis-lyqvhu-redis-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success get redis-sentinel replicas test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-redis-lyqvhu" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-redis-lyqvhu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-redis-lyqvhu namespace: ns-rxzwd spec: selector: namespaces: - ns-rxzwd labelSelectors: apps.kubeblocks.io/pod-name: redis-lyqvhu-redis-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-redis-lyqvhu.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-redis-lyqvhu created apply test-chaos-mesh-podkill-redis-lyqvhu.yaml Success `rm -rf test-chaos-mesh-podkill-redis-lyqvhu.yaml` check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd DoNotTerminate Updating Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:43 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-redis-lyqvhu --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-redis-lyqvhu" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-redis-lyqvhu patched check failover pod name failover pod name:redis-lyqvhu-redis-1 failover podkill Success `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: benchtest-redis-lyqvhu namespace: ns-rxzwd spec: containers: - name: test-benchmark imagePullPolicy: IfNotPresent image: docker.io/apecloud/redis-benchmark:latest args: - "-h" - "redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local" - "-p" - "6379" - "-a" - "wVrHR32762" - "-n" - "500" - "-c" - "4" - "-q" restartPolicy: Never `kubectl apply -f benchtest-redis-lyqvhu.yaml` pod/benchtest-redis-lyqvhu created apply benchtest-redis-lyqvhu.yaml Success check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 1/1 Running 0 11s check pod benchtest-redis-lyqvhu status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 0/1 Completed 0 16s `rm -rf benchtest-redis-lyqvhu.yaml` `kubectl logs benchtest-redis-lyqvhu --tail 30 --namespace ns-rxzwd ` PING_INLINE: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) PING_INLINE: 29411.76 requests per second, p50=0.095 msec PING_MBULK: 38461.54 requests per second, p50=0.095 msec SET: 26315.79 requests per second, p50=0.111 msec GET: 31250.00 requests per second, p50=0.095 msec INCR: 11904.76 requests per second, p50=0.127 msec LPUSH: rps=432.5 (overall: 2725.0) avg_msec=0.134 (overall: 0.134) LPUSH: 6410.26 requests per second, p50=0.135 msec RPUSH: 7936.51 requests per second, p50=0.111 msec LPOP: 27777.78 requests per second, p50=0.127 msec RPOP: 10000.00 requests per second, p50=0.119 msec SADD: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) SADD: 33333.34 requests per second, p50=0.103 msec HSET: 31250.00 requests per second, p50=0.111 msec SPOP: 29411.76 requests per second, p50=0.095 msec ZADD: 16129.03 requests per second, p50=0.095 msec ZPOPMIN: 38461.54 requests per second, p50=0.095 msec LPUSH (needed to benchmark LRANGE): 29411.76 requests per second, p50=0.119 msec LRANGE_100 (first 100 elements): rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) LRANGE_100 (first 100 elements): 25000.00 requests per second, p50=0.119 msec LRANGE_300 (first 300 elements): 9433.96 requests per second, p50=0.191 msec LRANGE_500 (first 500 elements): 5813.95 requests per second, p50=0.279 msec LRANGE_600 (first 600 elements): rps=458.2 (overall: 6052.6) avg_msec=0.333 (overall: 0.333) LRANGE_600 (first 600 elements): 7692.31 requests per second, p50=0.255 msec MSET (10 keys): 10204.08 requests per second, p50=0.127 msec XADD: 27777.78 requests per second, p50=0.119 msec `kubectl delete pod benchtest-redis-lyqvhu --force --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-redis-lyqvhu" force deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: benchtest-redis-lyqvhu namespace: ns-rxzwd spec: containers: - name: test-benchmark imagePullPolicy: IfNotPresent image: docker.io/apecloud/redis-benchmark:latest args: - "-h" - "redis-lyqvhu-redis-twemproxy-twemproxy.ns-rxzwd.svc.cluster.local" - "-p" - "22121" - "-a" - "wVrHR32762" - "-n" - "500" - "-c" - "4" - "-t" - "PING_MBULK,SET,GET,INCR,LPUSH,RPUSH,LPOP,RPOP,SADD,HSET,SPOP,ZADD,ZPOPMIN,LRANGE_100,LRANGE_300,LRANGE_500,LRANGE_600,MSET" - "-q" restartPolicy: Never `kubectl apply -f benchtest-redis-lyqvhu.yaml` pod/benchtest-redis-lyqvhu created apply benchtest-redis-lyqvhu.yaml Success check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 0/1 ContainerCreating 0 0s check pod benchtest-redis-lyqvhu status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 0/1 Completed 0 4s `rm -rf benchtest-redis-lyqvhu.yaml` `kubectl logs benchtest-redis-lyqvhu --tail 30 --namespace ns-rxzwd ` WARNING: Could not fetch server CONFIG PING_MBULK: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) PING_MBULK: 33333.34 requests per second, p50=0.103 msec SET: 14285.71 requests per second, p50=0.183 msec GET: 5952.38 requests per second, p50=0.151 msec INCR: rps=1410.4 (overall: 7224.5) avg_msec=0.185 (overall: 0.185) INCR: 6172.84 requests per second, p50=0.183 msec LPUSH: 7352.94 requests per second, p50=0.175 msec RPUSH: 5617.98 requests per second, p50=0.167 msec LPOP: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) LPOP: 6756.76 requests per second, p50=0.175 msec RPOP: 6944.44 requests per second, p50=0.191 msec SADD: 22727.27 requests per second, p50=0.151 msec HSET: rps=754.0 (overall: 9047.6) avg_msec=0.168 (overall: 0.168) HSET: 10869.57 requests per second, p50=0.167 msec SPOP: 6849.31 requests per second, p50=0.151 msec ZADD: 6944.44 requests per second, p50=0.143 msec ZPOPMIN: rps=340.0 (overall: 17000.0) avg_msec=0.149 (overall: 0.149) ZPOPMIN: 7246.38 requests per second, p50=0.143 msec LPUSH (needed to benchmark LRANGE): 7936.51 requests per second, p50=0.167 msec LRANGE_100 (first 100 elements): 19230.77 requests per second, p50=0.175 msec LRANGE_300 (first 300 elements): rps=43.8 (overall: 1000.0) avg_msec=0.252 (overall: 0.252) LRANGE_300 (first 300 elements): 3597.12 requests per second, p50=0.231 msec LRANGE_500 (first 500 elements): 7246.38 requests per second, p50=0.263 msec LRANGE_600 (first 600 elements): rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) LRANGE_600 (first 600 elements): 3246.75 requests per second, p50=0.247 msec MSET (10 keys): rps=1636.0 (overall: 5311.7) avg_msec=0.715 (overall: 0.715) MSET (10 keys): 6024.10 requests per second, p50=0.223 msec `kubectl delete pod benchtest-redis-lyqvhu --force --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-redis-lyqvhu" force deleted No resources found in ns-rxzwd namespace. `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: benchtest-redis-lyqvhu namespace: ns-rxzwd spec: containers: - name: test-benchmark imagePullPolicy: IfNotPresent image: docker.io/apecloud/redis-benchmark:latest args: - "-h" - "10.224.0.7" - "-p" - "6379" - "-a" - "wVrHR32762" - "-n" - "500" - "-c" - "4" - "-q" restartPolicy: Never `kubectl apply -f benchtest-redis-lyqvhu.yaml` pod/benchtest-redis-lyqvhu created apply benchtest-redis-lyqvhu.yaml Success check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 0/1 ContainerCreating 0 0s check pod benchtest-redis-lyqvhu status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-lyqvhu 0/1 Completed 0 5s `rm -rf benchtest-redis-lyqvhu.yaml` `kubectl logs benchtest-redis-lyqvhu --tail 30 --namespace ns-rxzwd ` PING_INLINE: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) PING_INLINE: 33333.34 requests per second, p50=0.095 msec PING_MBULK: 31250.00 requests per second, p50=0.095 msec SET: 26315.79 requests per second, p50=0.103 msec GET: 6097.56 requests per second, p50=0.087 msec INCR: 29411.76 requests per second, p50=0.111 msec LPUSH: 5882.35 requests per second, p50=0.103 msec RPUSH: 31250.00 requests per second, p50=0.103 msec LPOP: rps=0.0 (overall: 0.0) avg_msec=-nan (overall: -nan) LPOP: 5617.98 requests per second, p50=0.111 msec RPOP: 5555.56 requests per second, p50=0.119 msec SADD: 33333.34 requests per second, p50=0.095 msec HSET: rps=1824.7 (overall: 7896.6) avg_msec=0.109 (overall: 0.109) HSET: 5952.38 requests per second, p50=0.111 msec SPOP: 26315.79 requests per second, p50=0.095 msec ZADD: 33333.34 requests per second, p50=0.095 msec ZPOPMIN: 7246.38 requests per second, p50=0.095 msec LPUSH (needed to benchmark LRANGE): 26315.79 requests per second, p50=0.119 msec LRANGE_100 (first 100 elements): 5555.56 requests per second, p50=0.119 msec LRANGE_300 (first 300 elements): rps=692.0 (overall: 14416.7) avg_msec=0.158 (overall: 0.158) LRANGE_300 (first 300 elements): 5681.82 requests per second, p50=0.151 msec LRANGE_500 (first 500 elements): 10869.57 requests per second, p50=0.223 msec LRANGE_600 (first 600 elements): 5000.00 requests per second, p50=0.263 msec MSET (10 keys): rps=12.0 (overall: 111.1) avg_msec=0.297 (overall: 0.297) MSET (10 keys): 7462.69 requests per second, p50=0.119 msec XADD: 5208.33 requests per second, p50=0.111 msec `kubectl delete pod benchtest-redis-lyqvhu --force --namespace ns-rxzwd ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-redis-lyqvhu" force deleted cluster update terminationPolicy WipeOut `kbcli cluster update redis-lyqvhu --termination-policy=WipeOut --namespace ns-rxzwd ` cluster.apps.kubeblocks.io/redis-lyqvhu updated check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd WipeOut Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:43 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done cluster datafile backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-bxwnt -n kb-hmwej -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-bxwnt -n kb-hmwej -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-hmwej get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-hmwej -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-hmwej -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-hmwej.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-hmwej.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-669d87677b-7sjlg --namespace kb-hmwej -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup redis-lyqvhu --method datafile --namespace ns-rxzwd ` Backup backup-ns-rxzwd-redis-lyqvhu-20250901125006 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-rxzwd-redis-lyqvhu-20250901125006 -n ns-rxzwd check backup status `kbcli cluster list-backups redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-rxzwd-redis-lyqvhu-20250901125006 ns-rxzwd redis-lyqvhu datafile Running Sep 01,2025 12:50 UTC+0800 backup_status:redis-lyqvhu-datafile-Running backup_status:redis-lyqvhu-datafile-Running backup_status:redis-lyqvhu-datafile-Running check backup status done backup_status:backup-ns-rxzwd-redis-lyqvhu-20250901125006 ns-rxzwd redis-lyqvhu datafile Completed 17491 16s Sep 01,2025 12:50 UTC+0800 Sep 01,2025 12:50 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "redis-lyqvhu-backup" not found `kbcli cluster describe-backup backup-ns-rxzwd-redis-lyqvhu-20250901125006 --namespace ns-rxzwd ` Name: backup-ns-rxzwd-redis-lyqvhu-20250901125006 Cluster: redis-lyqvhu Namespace: ns-rxzwd Spec: Method: datafile Policy Name: redis-lyqvhu-redis-backup-policy Status: Phase: Completed Total Size: 17491 ActionSet Name: redis-physical-backup Repository: backuprepo-kbcli-test Duration: 16s Start Time: Sep 01,2025 12:50 UTC+0800 Completion Time: Sep 01,2025 12:50 UTC+0800 Path: /ns-rxzwd/redis-lyqvhu-5ebc8fd3-2f93-4b41-bf25-b87e3c73d8e5/redis/backup-ns-rxzwd-redis-lyqvhu-20250901125006 Warning Events: `kbcli cluster restore redis-lyqvhu-backup --backup backup-ns-rxzwd-redis-lyqvhu-20250901125006 --namespace ns-rxzwd ` Cluster redis-lyqvhu-backup created check cluster status `kbcli cluster list redis-lyqvhu-backup --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu-backup ns-rxzwd WipeOut Sep 01,2025 12:50 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu-backup --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-backup-redis-0 ns-rxzwd redis-lyqvhu-backup redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-1 ns-rxzwd redis-lyqvhu-backup redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-sentinel-0 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-sentinel-1 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-sentinel-2 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-0 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-1 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:50 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-2 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-backup-redis-0;secondary: redis-lyqvhu-backup-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu-backup` set secret: redis-lyqvhu-backup-redis-account-default `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-backup-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-backup-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kbcli cluster describe-backup backup-ns-rxzwd-redis-lyqvhu-20250901125006 --namespace ns-rxzwd ` Name: backup-ns-rxzwd-redis-lyqvhu-20250901125006 Cluster: redis-lyqvhu Namespace: ns-rxzwd Spec: Method: datafile Policy Name: redis-lyqvhu-redis-backup-policy Status: Phase: Completed Total Size: 17491 ActionSet Name: redis-physical-backup Repository: backuprepo-kbcli-test Duration: 16s Start Time: Sep 01,2025 12:50 UTC+0800 Completion Time: Sep 01,2025 12:50 UTC+0800 Path: /ns-rxzwd/redis-lyqvhu-5ebc8fd3-2f93-4b41-bf25-b87e3c73d8e5/redis/backup-ns-rxzwd-redis-lyqvhu-20250901125006 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu-backup` set secret: redis-lyqvhu-backup-redis-account-default `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.4.5 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:b6b2e66014f8e765 redis_mode:standalone os:Linux 5.15.0-1092-azure x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:3c26f408924f0fb33453870b91ae00f92625d79c tcp_port:6379 server_time_usec:1756702352294703 uptime_in_seconds:50 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:11871888 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo 'echo "CONFIG GET maxclients" | redis-cli -h redis-lyqvhu-backup-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-backup-redis-0 --namespace ns-rxzwd -- bash ` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 delete cluster redis-lyqvhu-backup `kbcli cluster delete redis-lyqvhu-backup --auto-approve --namespace ns-rxzwd ` Cluster redis-lyqvhu-backup deleted pod_info:redis-lyqvhu-backup-redis-0 3/3 Running 0 106s redis-lyqvhu-backup-redis-1 3/3 Running 0 106s redis-lyqvhu-backup-redis-sentinel-0 2/2 Running 0 2m9s redis-lyqvhu-backup-redis-sentinel-1 2/2 Running 0 2m9s redis-lyqvhu-backup-redis-sentinel-2 2/2 Running 0 2m10s redis-lyqvhu-backup-redis-twemproxy-0 1/1 Running 0 2m9s redis-lyqvhu-backup-redis-twemproxy-1 1/1 Running 0 97s redis-lyqvhu-backup-redis-twemproxy-2 1/1 Running 0 75s pod_info:redis-lyqvhu-backup-redis-twemproxy-0 1/1 Terminating 0 2m30s redis-lyqvhu-backup-redis-twemproxy-1 1/1 Terminating 0 118s redis-lyqvhu-backup-redis-twemproxy-2 1/1 Terminating 0 96s No resources found in ns-rxzwd namespace. delete cluster pod done No resources found in ns-rxzwd namespace. check cluster resource non-exist OK: pvc No resources found in ns-rxzwd namespace. delete cluster done No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. cluster rebulid instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: redis-lyqvhu-rebuildinstance- namespace: ns-rxzwd spec: type: RebuildInstance clusterName: redis-lyqvhu force: true rebuildFrom: - componentName: redis instances: - name: redis-lyqvhu-redis-0 backupName: backup-ns-rxzwd-redis-lyqvhu-20250901125006 inPlace: false check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_redis-lyqvhu.yaml` opsrequest.apps.kubeblocks.io/redis-lyqvhu-rebuildinstance-jgnzt created create test_ops_cluster_redis-lyqvhu.yaml Success `rm -rf test_ops_cluster_redis-lyqvhu.yaml` check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` No opsrequests found in ns-rxzwd namespace. No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: No opsrequests found in ns-rxzwd namespace. ops_status: not found ops rebuildinstance check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd WipeOut Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:43 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-rxzwd-redis-lyqvhu-20250901125006 --namespace ns-rxzwd ` backup.dataprotection.kubeblocks.io/backup-ns-rxzwd-redis-lyqvhu-20250901125006 patched `kbcli cluster delete-backup redis-lyqvhu --name backup-ns-rxzwd-redis-lyqvhu-20250901125006 --force --auto-approve --namespace ns-rxzwd ` Backup backup-ns-rxzwd-redis-lyqvhu-20250901125006 deleted No opsrequests found in ns-rxzwd namespace. `kubectl get backupschedule -l app.kubernetes.io/instance=redis-lyqvhu ` `kubectl get backupschedule redis-lyqvhu-redis-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched check backup status `kbcli cluster list-backups redis-lyqvhu --namespace ns-rxzwd ` No backups found in ns-rxzwd namespace. No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- No backups found in ns-rxzwd namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=redis-lyqvhu ` `kubectl get backupschedule redis-lyqvhu-redis-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backup_status:redis-lyqvhu-datafile-Running check backup status done backup_status:redis-lyqvhu-datafile-20250901045800 ns-rxzwd redis-lyqvhu datafile Completed 17544 10s Sep 01,2025 12:58 UTC+0800 Sep 01,2025 12:58 UTC+0800 Sep 08,2025 12:58 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=redis-lyqvhu ` `kubectl get backupschedule redis-lyqvhu-redis-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-lyqvhu-redis-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "redis-lyqvhu-backup" not found `kbcli cluster describe-backup redis-lyqvhu-datafile-20250901045800 --namespace ns-rxzwd ` Name: redis-lyqvhu-datafile-20250901045800 Cluster: redis-lyqvhu Namespace: ns-rxzwd Spec: Method: datafile Policy Name: redis-lyqvhu-redis-backup-policy Status: Phase: Completed Total Size: 17544 ActionSet Name: redis-physical-backup Repository: backuprepo-kbcli-test Duration: 10s Expiration Time: Sep 08,2025 12:58 UTC+0800 Start Time: Sep 01,2025 12:58 UTC+0800 Completion Time: Sep 01,2025 12:58 UTC+0800 Path: /ns-rxzwd/redis-lyqvhu-5ebc8fd3-2f93-4b41-bf25-b87e3c73d8e5/redis/redis-lyqvhu-datafile-20250901045800 Warning Events: `kbcli cluster restore redis-lyqvhu-backup --backup redis-lyqvhu-datafile-20250901045800 --namespace ns-rxzwd ` Cluster redis-lyqvhu-backup created check cluster status `kbcli cluster list redis-lyqvhu-backup --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu-backup ns-rxzwd WipeOut Sep 01,2025 12:58 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu-backup --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-backup-redis-0 ns-rxzwd redis-lyqvhu-backup redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-1 ns-rxzwd redis-lyqvhu-backup redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-sentinel-0 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-sentinel-1 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-sentinel-2 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-0 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-1 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 12:58 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-2 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:59 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-backup-redis-0;secondary: redis-lyqvhu-backup-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu-backup` set secret: redis-lyqvhu-backup-redis-account-default `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-backup-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-backup-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kbcli cluster describe-backup redis-lyqvhu-datafile-20250901045800 --namespace ns-rxzwd ` Name: redis-lyqvhu-datafile-20250901045800 Cluster: redis-lyqvhu Namespace: ns-rxzwd Spec: Method: datafile Policy Name: redis-lyqvhu-redis-backup-policy Status: Phase: Completed Total Size: 17544 ActionSet Name: redis-physical-backup Repository: backuprepo-kbcli-test Duration: 10s Expiration Time: Sep 08,2025 12:58 UTC+0800 Start Time: Sep 01,2025 12:58 UTC+0800 Completion Time: Sep 01,2025 12:58 UTC+0800 Path: /ns-rxzwd/redis-lyqvhu-5ebc8fd3-2f93-4b41-bf25-b87e3c73d8e5/redis/redis-lyqvhu-datafile-20250901045800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu-backup` set secret: redis-lyqvhu-backup-redis-account-default `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.4.5 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:b6b2e66014f8e765 redis_mode:standalone os:Linux 5.15.0-1092-azure x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:54880c997796e1eec7b82a2c7b22c03b68a1a00e tcp_port:6379 server_time_usec:1756702808076098 uptime_in_seconds:76 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:11872344 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo 'echo "CONFIG GET maxclients" | redis-cli -h redis-lyqvhu-backup-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-backup-redis-0 --namespace ns-rxzwd -- bash ` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 delete cluster redis-lyqvhu-backup `kbcli cluster delete redis-lyqvhu-backup --auto-approve --namespace ns-rxzwd ` Cluster redis-lyqvhu-backup deleted pod_info:redis-lyqvhu-backup-redis-0 3/3 Running 0 85s redis-lyqvhu-backup-redis-1 3/3 Running 0 85s redis-lyqvhu-backup-redis-sentinel-0 2/2 Running 0 107s redis-lyqvhu-backup-redis-sentinel-1 2/2 Running 0 107s redis-lyqvhu-backup-redis-sentinel-2 2/2 Running 0 108s redis-lyqvhu-backup-redis-twemproxy-0 1/1 Running 0 107s redis-lyqvhu-backup-redis-twemproxy-1 1/1 Running 0 77s redis-lyqvhu-backup-redis-twemproxy-2 1/1 Running 0 54s pod_info:redis-lyqvhu-backup-redis-twemproxy-0 1/1 Terminating 0 2m8s redis-lyqvhu-backup-redis-twemproxy-1 1/1 Terminating 0 98s redis-lyqvhu-backup-redis-twemproxy-2 1/1 Terminating 0 75s No resources found in ns-rxzwd namespace. delete cluster pod done No resources found in ns-rxzwd namespace. check cluster resource non-exist OK: pvc No resources found in ns-rxzwd namespace. delete cluster done No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups redis-lyqvhu-datafile-20250901045800 --namespace ns-rxzwd ` backup.dataprotection.kubeblocks.io/redis-lyqvhu-datafile-20250901045800 patched `kbcli cluster delete-backup redis-lyqvhu --name redis-lyqvhu-datafile-20250901045800 --force --auto-approve --namespace ns-rxzwd ` Backup redis-lyqvhu-datafile-20250901045800 deleted No opsrequests found in ns-rxzwd namespace. check cluster storage class cluster volume-snapshot backup `kbcli cluster backup redis-lyqvhu --method volume-snapshot --namespace ns-rxzwd ` Backup backup-ns-rxzwd-redis-lyqvhu-20250901130101 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-rxzwd-redis-lyqvhu-20250901130101 -n ns-rxzwd check backup status `kbcli cluster list-backups redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-rxzwd-redis-lyqvhu-20250901130101 ns-rxzwd redis-lyqvhu volume-snapshot Running Sep 01,2025 13:01 UTC+0800 backup_status:redis-lyqvhu-volume-snapshot-Running backup_status:redis-lyqvhu-volume-snapshot-Running backup_status:redis-lyqvhu-volume-snapshot-Running check backup status done backup_status:backup-ns-rxzwd-redis-lyqvhu-20250901130101 ns-rxzwd redis-lyqvhu volume-snapshot Completed 4Gi 13s Sep 01,2025 13:01 UTC+0800 Sep 01,2025 13:01 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "redis-lyqvhu-backup" not found `kbcli cluster describe-backup backup-ns-rxzwd-redis-lyqvhu-20250901130101 --namespace ns-rxzwd ` Name: backup-ns-rxzwd-redis-lyqvhu-20250901130101 Cluster: redis-lyqvhu Namespace: ns-rxzwd Spec: Method: volume-snapshot Policy Name: redis-lyqvhu-redis-backup-policy Status: Phase: Completed Total Size: 4Gi Duration: 13s Start Time: Sep 01,2025 13:01 UTC+0800 Completion Time: Sep 01,2025 13:01 UTC+0800 Time Range Start: Sep 01,2025 13:01 UTC+0800 Time Range End: Sep 01,2025 13:01 UTC+0800 Warning Events: `kbcli cluster restore redis-lyqvhu-backup --backup backup-ns-rxzwd-redis-lyqvhu-20250901130101 --namespace ns-rxzwd ` Cluster redis-lyqvhu-backup created check cluster status `kbcli cluster list redis-lyqvhu-backup --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu-backup ns-rxzwd WipeOut Sep 01,2025 13:01 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu-backup --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-backup-redis-0 ns-rxzwd redis-lyqvhu-backup redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-1 ns-rxzwd redis-lyqvhu-backup redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-sentinel-0 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-sentinel-1 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-sentinel-2 ns-rxzwd redis-lyqvhu-backup redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-0 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-1 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000008/10.224.0.9 Sep 01,2025 13:01 UTC+0800 redis-lyqvhu-backup-redis-twemproxy-2 ns-rxzwd redis-lyqvhu-backup redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 13:02 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-backup-redis-0;secondary: redis-lyqvhu-backup-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu-backup` set secret: redis-lyqvhu-backup-redis-account-default `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-backup-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-backup-redis-0 --namespace ns-rxzwd -- bash` check cluster connect done `kbcli cluster describe-backup backup-ns-rxzwd-redis-lyqvhu-20250901130101 --namespace ns-rxzwd ` Name: backup-ns-rxzwd-redis-lyqvhu-20250901130101 Cluster: redis-lyqvhu Namespace: ns-rxzwd Spec: Method: volume-snapshot Policy Name: redis-lyqvhu-redis-backup-policy Status: Phase: Completed Total Size: 4Gi Duration: 13s Start Time: Sep 01,2025 13:01 UTC+0800 Completion Time: Sep 01,2025 13:01 UTC+0800 Time Range Start: Sep 01,2025 13:01 UTC+0800 Time Range End: Sep 01,2025 13:01 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu-backup` set secret: redis-lyqvhu-backup-redis-account-default `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.4.5 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:b6b2e66014f8e765 redis_mode:standalone os:Linux 5.15.0-1092-azure x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:dd10a3b220b24dc666b647787986fdd3f9901304 tcp_port:6379 server_time_usec:1756702964201796 uptime_in_seconds:50 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:11872500 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo 'echo "CONFIG GET maxclients" | redis-cli -h redis-lyqvhu-backup-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-backup-redis-0 --namespace ns-rxzwd -- bash ` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 delete cluster redis-lyqvhu-backup `kbcli cluster delete redis-lyqvhu-backup --auto-approve --namespace ns-rxzwd ` Cluster redis-lyqvhu-backup deleted pod_info:redis-lyqvhu-backup-redis-0 3/3 Running 0 85s redis-lyqvhu-backup-redis-1 3/3 Running 0 85s redis-lyqvhu-backup-redis-sentinel-0 2/2 Running 0 85s redis-lyqvhu-backup-redis-sentinel-1 2/2 Running 0 85s redis-lyqvhu-backup-redis-sentinel-2 2/2 Running 0 86s redis-lyqvhu-backup-redis-twemproxy-0 1/1 Running 0 85s redis-lyqvhu-backup-redis-twemproxy-1 1/1 Running 0 67s redis-lyqvhu-backup-redis-twemproxy-2 1/1 Running 0 39s pod_info:redis-lyqvhu-backup-redis-twemproxy-0 1/1 Terminating 0 106s redis-lyqvhu-backup-redis-twemproxy-1 1/1 Terminating 0 88s redis-lyqvhu-backup-redis-twemproxy-2 1/1 Terminating 0 60s No resources found in ns-rxzwd namespace. delete cluster pod done No resources found in ns-rxzwd namespace. check cluster resource non-exist OK: pvc No resources found in ns-rxzwd namespace. delete cluster done No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-rxzwd-redis-lyqvhu-20250901130101 --namespace ns-rxzwd ` backup.dataprotection.kubeblocks.io/backup-ns-rxzwd-redis-lyqvhu-20250901130101 patched `kbcli cluster delete-backup redis-lyqvhu --name backup-ns-rxzwd-redis-lyqvhu-20250901130101 --force --auto-approve --namespace ns-rxzwd ` Backup backup-ns-rxzwd-redis-lyqvhu-20250901130101 deleted No opsrequests found in ns-rxzwd namespace. cluster list-logs `kbcli cluster list-logs redis-lyqvhu --namespace ns-rxzwd ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update redis-lyqvhu --enable-all-logs=true --namespace ns-rxzwd Error from server (NotFound): pods "redis-lyqvhu-redis-1" not found cluster logs `kbcli cluster logs redis-lyqvhu --tail 30 --namespace ns-rxzwd ` Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) + echo 'replicaof redis-lyqvhu-redis-0.redis-lyqvhu-redis-headless.ns-rxzwd.svc.cluster.local 6379' + rebuild_redis_acl_file + '[' -f /data/users.acl ']' + sed -i '/user default on/d' /data/users.acl + sed -i '/user kbreplicator on/d' /data/users.acl + sed -i '/user kbreplicator-sentinel on/d' /data/users.acl + build_redis_default_accounts + set +x + echo 'aclfile /data/users.acl' + echo 'build default accounts succeeded!' build default accounts succeeded! + start_redis_server + module_path=/opt/redis-stack/lib + [[ '' == \t\r\u\e ]] + exec_cmd='exec redis-server /etc/redis/redis.conf' + '[' -f /opt/redis-stack/lib/redisearch.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so ' + '[' -f /opt/redis-stack/lib/redistimeseries.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so ' + '[' -f /opt/redis-stack/lib/rejson.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so ' + '[' -f /opt/redis-stack/lib/redisbloom.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so ' + '[' -f /opt/redis-stack/lib/redisgraph.so ']' + '[' -f /opt/redis-stack/lib/rediscompat.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so' + echo 'Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so' + eval 'exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so' ++ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so cluster logs running `kbcli cluster logs redis-lyqvhu --tail 30 --file-type=running --namespace ns-rxzwd ` error: get pod component definition name in cluster.yaml fail cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose redis-lyqvhu --auto-approve --force=true --type vpc --enable false --components redis --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-expose-9mq5s created successfully, you can view the progress: kbcli cluster describe-ops redis-lyqvhu-expose-9mq5s -n ns-rxzwd check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-expose-9mq5s ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 13:03 UTC+0800 check cluster status `kbcli cluster list redis-lyqvhu --show-labels --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-lyqvhu ns-rxzwd WipeOut Running Sep 01,2025 11:24 UTC+0800 app.kubernetes.io/instance=redis-lyqvhu check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-lyqvhu --namespace ns-rxzwd ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-lyqvhu-redis-0 ns-rxzwd redis-lyqvhu redis Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000006/10.224.0.10 Sep 01,2025 12:43 UTC+0800 redis-lyqvhu-redis-1 ns-rxzwd redis-lyqvhu redis Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:17 UTC+0800 redis-lyqvhu-redis-sentinel-0 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-sentinel-1 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-sentinel-2 ns-rxzwd redis-lyqvhu redis-sentinel Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-default-25371978-vmss000000/10.224.0.4 Sep 01,2025 12:27 UTC+0800 redis-lyqvhu-redis-twemproxy-0 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:11 UTC+0800 redis-lyqvhu-redis-twemproxy-1 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:10 UTC+0800 redis-lyqvhu-redis-twemproxy-2 ns-rxzwd redis-lyqvhu redis-twemproxy Running 0 100m / 100m 512Mi / 512Mi data:1Gi aks-cicdamdpool-38368475-vmss000000/10.224.0.5 Sep 01,2025 12:27 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-lyqvhu-redis-1;secondary: redis-lyqvhu-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-lyqvhu-redis-redis.ns-rxzwd.svc.cluster.local -p 6379 -a wVrHR32762' | kubectl exec -it redis-lyqvhu-redis-1 --namespace ns-rxzwd -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-lyqvhu --status all --namespace ns-rxzwd ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-lyqvhu-expose-9mq5s ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 13:03 UTC+0800 ops_status:redis-lyqvhu-expose-9mq5s ns-rxzwd Expose redis-lyqvhu redis Running 0/1 Sep 01,2025 13:03 UTC+0800 check ops status done ops_status:redis-lyqvhu-expose-9mq5s ns-rxzwd Expose redis-lyqvhu redis Succeed 1/1 Sep 01,2025 13:03 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests redis-lyqvhu-expose-9mq5s --namespace ns-rxzwd ` opsrequest.apps.kubeblocks.io/redis-lyqvhu-expose-9mq5s patched `kbcli cluster delete-ops --name redis-lyqvhu-expose-9mq5s --force --auto-approve --namespace ns-rxzwd ` OpsRequest redis-lyqvhu-expose-9mq5s deleted `echo 'get mykey' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check data: Connect to instance redis-lyqvhu-redis-1: out of redis-lyqvhu-redis-1(primary), redis-lyqvhu-redis-0(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. fuqur check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly data: Defaulted container "redis" out of: redis, lorry, config-manager, init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file fuqur Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success check db_client batch data count `echo 'get executions_loop_key' | kbcli cluster connect redis-lyqvhu --namespace ns-rxzwd ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-lyqvhu` set secret: redis-lyqvhu-redis-account-default `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-lyqvhu-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:wVrHR32762;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a wVrHR32762 " | kubectl exec -it redis-lyqvhu-redis-0 --namespace ns-rxzwd -- bash ` check readonly db_client batch data Success delete cluster redis-lyqvhu `kbcli cluster delete redis-lyqvhu --auto-approve --namespace ns-rxzwd ` Cluster redis-lyqvhu deleted pod_info:redis-lyqvhu-redis-0 3/3 Running 0 20m redis-lyqvhu-redis-1 3/3 Running 0 46m redis-lyqvhu-redis-sentinel-0 2/2 Running 0 52m redis-lyqvhu-redis-sentinel-1 2/2 Running 0 53m redis-lyqvhu-redis-sentinel-2 2/2 Running 0 36m redis-lyqvhu-redis-twemproxy-0 1/1 Running 0 52m redis-lyqvhu-redis-twemproxy-1 1/1 Running 0 53m redis-lyqvhu-redis-twemproxy-2 1/1 Running 0 36m pod_info:redis-lyqvhu-redis-twemproxy-0 1/1 Terminating 0 52m redis-lyqvhu-redis-twemproxy-1 1/1 Terminating 0 54m redis-lyqvhu-redis-twemproxy-2 1/1 Terminating 0 36m No resources found in ns-rxzwd namespace. delete cluster pod done No resources found in ns-rxzwd namespace. check cluster resource non-exist OK: pvc No resources found in ns-rxzwd namespace. delete cluster done No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. No resources found in ns-rxzwd namespace. Redis Test Suite All Done! --------------------------------------Redis (Topology = replication Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=redis;ClusterVersion=redis-7.2.7;]|[Description=Create a cluster with the specified cluster definition redis and cluster version redis-7.2.7] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=fuqur]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=fuqur;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Expose]|[Enable=true;TYPE=vpc;ComponentName=redis]|[Description=Expose Enable the vpc service with redis component] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=redis]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [SKIPPED]|[No-Failover]|[HA=Evicting Pod;ComponentName=redis]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[VerticalScaling]|[ComponentName=redis]|[Description=VerticalScaling the cluster specify component redis] [PASSED]|[Reconfiguring]|[ComponentName=redis;maxclients=10001]|[Description=Reconfiguring the cluster specify component redis set maxclients=10001] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=redis]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=redis]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=redis]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=redis]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=redis]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [WARNING]|[CheckFailover]|[FailoverType=networkdelay]|[Description=-] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=redis]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=redis]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=redis]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[VolumeExpansion]|[ComponentName=redis]|[Description=VolumeExpansion the cluster specify component redis] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[HorizontalScaling Out]|[ComponentName=redis]|[Description=HorizontalScaling Out the cluster specify component redis] [PASSED]|[HorizontalScaling In]|[ComponentName=redis]|[Description=HorizontalScaling In the cluster specify component redis] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Pod Kill;ComponentName=redis]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [PASSED]|[Bench]|[ComponentName=redis]|[Description=Bench the cluster service with redis component] [PASSED]|[Bench]|[Component redis-twemproxy]|[Description=Bench the cluster service] [PASSED]|[Bench]|[HostType=LB;ComponentName=redis]|[Description=Bench the cluster LB service with redis component] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=datafile]|[Description=The cluster datafile Backup] [PASSED]|[Restore]|[BackupMethod=datafile]|[Description=The cluster datafile Restore] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=datafile]|[Description=Delete the datafile restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=redis]|[Description=Rebuild the cluster instance specify component redis] [PASSED]|[Backup]|[Schedule=true;BackupMethod=datafile]|[Description=The cluster Schedule datafile Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=datafile]|[Description=The cluster Schedule datafile Restore] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=datafile]|[Description=Delete the Schedule datafile restore cluster] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Expose]|[Disable=true;TYPE=vpc;ComponentName=redis]|[Description=Expose Disable the vpc service with redis component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]