source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-rdwsc ` `kubectl create namespace ns-rdwsc` namespace/ns-rdwsc created create namespace ns-rdwsc done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.0` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 56.9M 0 --:--:-- --:--:-- --:--:-- 56.9M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.0 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Kubernetes Env: v1.32.5-eks-5d4a308 check snapshot controller check snapshot controller done eks default-vsc found POD_RESOURCES: No resources found found default storage class: gp3 KubeBlocks version is:1.0.0 skip upgrade KubeBlocks current KubeBlocks version: 1.0.0 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:redis set component version set component version:redis set service versions:7.2.7,7.2.4,7.0.6 set service versions sorted:7.0.6,7.2.4,7.2.7 set redis component definition set redis component definition redis-cluster-7-1.0.0-alpha.0 set replicas first:2,7.0.6|2,7.2.4|2,7.2.7 set replicas third:2,7.2.7 set replicas fourth:2,7.2.4 set minimum cmpv service version set minimum cmpv service version replicas:2,7.2.4 REPORT_COUNT:1 CLUSTER_TOPOLOGY:replication topology replication found in cluster definition redis set redis component definition set redis component definition redis-7-1.0.0-alpha.0 LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-rdwsc namespace. termination_policy:WipeOut create 2 replica WipeOut redis cluster check component definition set component definition by component version check cmpd by labels set component definition1: redis-7-1.0.0-alpha.0 by component version:redis apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: redis-tqnwum namespace: ns-rdwsc spec: clusterDef: redis topology: replication terminationPolicy: WipeOut componentSpecs: - name: redis serviceVersion: 7.2.4 replicas: 2 disableExporter: false resources: limits: cpu: 100m memory: 0.5Gi requests: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: redis-sentinel serviceVersion: 7.2.4 replicas: 3 resources: limits: cpu: 100m memory: 0.5Gi requests: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi `kubectl apply -f test_create_redis-tqnwum.yaml` cluster.apps.kubeblocks.io/redis-tqnwum created apply test_create_redis-tqnwum.yaml Success `rm -rf test_create_redis-tqnwum.yaml` check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Creating May 28,2025 11:36 UTC+0800 clusterdefinition.kubeblocks.io/name=redis cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check pod redis-tqnwum-redis-0 container_name redis exist password 3L7022eJcr check pod redis-tqnwum-redis-0 container_name metrics exist password 3L7022eJcr check pod redis-tqnwum-redis-0 container_name kbagent exist password 3L7022eJcr No container logs contain secret password. describe cluster `kbcli cluster describe redis-tqnwum --namespace ns-rdwsc ` Name: redis-tqnwum Created Time: May 28,2025 11:36 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-rdwsc redis replication Running WipeOut Endpoints: COMPONENT INTERNAL EXTERNAL redis redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local:6379 redis-sentinel redis-tqnwum-redis-sentinel-redis-sentinel.ns-rdwsc.svc.cluster.local:26379 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME redis 7.2.4 redis-tqnwum-redis-0 primary Running us-west-2a ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis 7.2.4 redis-tqnwum-redis-1 secondary Running us-west-2a ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-sentinel 7.2.4 redis-tqnwum-redis-sentinel-0 Running us-west-2a ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-sentinel 7.2.4 redis-tqnwum-redis-sentinel-1 Running us-west-2a ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-sentinel 7.2.4 redis-tqnwum-redis-sentinel-2 Running us-west-2a ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS redis 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc redis-sentinel 100m / 100m 512Mi / 512Mi data:1Gi kb-default-sc Images: COMPONENT COMPONENT-DEFINITION IMAGE redis redis-7-1.0.0-alpha.0 docker.io/apecloud/redis-stack-server:7.2.0-v10 docker.io/apecloud/agamotto:0.1.2-beta.1 docker.io/apecloud/redis-stack-server:7.2.0-v14 redis-sentinel redis-sentinel-7-1.0.0-alpha.0 docker.io/apecloud/redis-stack-server:7.2.0-v10 docker.io/apecloud/redis-stack-server:7.2.0-v14 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-rdwsc redis-tqnwum `kbcli cluster label redis-tqnwum app.kubernetes.io/instance- --namespace ns-rdwsc ` label "app.kubernetes.io/instance" not found. `kbcli cluster label redis-tqnwum app.kubernetes.io/instance=redis-tqnwum --namespace ns-rdwsc ` `kbcli cluster label redis-tqnwum --list --namespace ns-rdwsc ` NAME NAMESPACE LABELS redis-tqnwum ns-rdwsc app.kubernetes.io/instance=redis-tqnwum clusterdefinition.kubeblocks.io/name=redis label cluster app.kubernetes.io/instance=redis-tqnwum Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=redis-tqnwum --namespace ns-rdwsc ` `kbcli cluster label redis-tqnwum --list --namespace ns-rdwsc ` NAME NAMESPACE LABELS redis-tqnwum ns-rdwsc app.kubernetes.io/instance=redis-tqnwum case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=redis label cluster case.name=kbcli.test1 Success `kbcli cluster label redis-tqnwum case.name=kbcli.test2 --overwrite --namespace ns-rdwsc ` `kbcli cluster label redis-tqnwum --list --namespace ns-rdwsc ` NAME NAMESPACE LABELS redis-tqnwum ns-rdwsc app.kubernetes.io/instance=redis-tqnwum case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=redis label cluster case.name=kbcli.test2 Success `kbcli cluster label redis-tqnwum case.name- --namespace ns-rdwsc ` `kbcli cluster label redis-tqnwum --list --namespace ns-rdwsc ` NAME NAMESPACE LABELS redis-tqnwum ns-rdwsc app.kubernetes.io/instance=redis-tqnwum clusterdefinition.kubeblocks.io/name=redis delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.2.4 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:3a23a5ac67fe7008 redis_mode:standalone os:Linux 5.10.236-228.935.amzn2.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:cee29154873e8c12bc6cdbd5183ad8b396139dc9 tcp_port:6379 server_time_usec:1748403522564634 uptime_in_seconds:83 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:3573058 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo " echo \"CONFIG GET maxclients\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-redis-tqnwum" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-redis-tqnwum --namespace ns-rdwsc ` Error from server (NotFound): pods "test-db-client-executionloop-redis-tqnwum" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-redis-tqnwum" not found `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-redis-tqnwum namespace: ns-rdwsc spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local" - "--user" - "default" - "--password" - "3L7022eJcr" - "--port" - "6379" - "--dbtype" - "redis" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-redis-tqnwum.yaml` pod/test-db-client-executionloop-redis-tqnwum created apply test-db-client-executionloop-redis-tqnwum.yaml Success `rm -rf test-db-client-executionloop-redis-tqnwum.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 39s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 62s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 1/1 Running 0 68s check pod test-db-client-executionloop-redis-tqnwum status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-redis-tqnwum 0/1 Completed 0 74s check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done --host redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local --user default --password 3L7022eJcr --port 6379 --dbtype redis --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Execution loop start: Execution loop start:set executions_loop_key executions_loop_test_1 [ 1s ] executions total: 254 successful: 254 failed: 0 disconnect: 0 [ 2s ] executions total: 556 successful: 556 failed: 0 disconnect: 0 [2025-05-28 03:39:07.374] Connection error occurred! [ 3s ] executions total: 801 successful: 800 failed: 1 disconnect: 1 [2025-05-28 03:39:07.374] Connection error occurred! [2025-05-28 03:39:08.379] Connection successfully recovered! The connection was restored in 1005 milliseconds. [ 4s ] executions total: 802 successful: 801 failed: 1 disconnect: 1 [ 5s ] executions total: 1156 successful: 1155 failed: 1 disconnect: 1 [ 6s ] executions total: 1466 successful: 1465 failed: 1 disconnect: 1 [2025-05-28 03:39:13.860] Connection error occurred! [ 7s ] executions total: 1602 successful: 1600 failed: 2 disconnect: 2 [2025-05-28 03:39:13.860] Connection error occurred! [2025-05-28 03:39:14.864] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 8s ] executions total: 1603 successful: 1601 failed: 2 disconnect: 2 [ 9s ] executions total: 1939 successful: 1937 failed: 2 disconnect: 2 [ 10s ] executions total: 2211 successful: 2209 failed: 2 disconnect: 2 [2025-05-28 03:39:20.473] Connection error occurred! [ 11s ] executions total: 2403 successful: 2400 failed: 3 disconnect: 3 [2025-05-28 03:39:20.473] Connection error occurred! [2025-05-28 03:39:21.477] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 12s ] executions total: 2404 successful: 2401 failed: 3 disconnect: 3 [ 13s ] executions total: 2757 successful: 2754 failed: 3 disconnect: 3 [ 14s ] executions total: 3079 successful: 3076 failed: 3 disconnect: 3 [2025-05-28 03:39:26.867] Connection error occurred! [ 15s ] executions total: 3204 successful: 3200 failed: 4 disconnect: 4 [2025-05-28 03:39:26.867] Connection error occurred! [2025-05-28 03:39:27.870] Connection successfully recovered! The connection was restored in 1003 milliseconds. [ 16s ] executions total: 3205 successful: 3201 failed: 4 disconnect: 4 [ 17s ] executions total: 3573 successful: 3569 failed: 4 disconnect: 4 [ 18s ] executions total: 3823 successful: 3819 failed: 4 disconnect: 4 [2025-05-28 03:39:33.365] Connection error occurred! [ 19s ] executions total: 4005 successful: 4000 failed: 5 disconnect: 5 [2025-05-28 03:39:33.365] Connection error occurred! [2025-05-28 03:39:34.375] Connection successfully recovered! The connection was restored in 1010 milliseconds. [ 20s ] executions total: 4006 successful: 4001 failed: 5 disconnect: 5 [ 21s ] executions total: 4359 successful: 4354 failed: 5 disconnect: 5 [ 22s ] executions total: 4620 successful: 4615 failed: 5 disconnect: 5 [2025-05-28 03:39:40.057] Connection error occurred! [ 23s ] executions total: 4806 successful: 4800 failed: 6 disconnect: 6 [2025-05-28 03:39:40.057] Connection error occurred! [2025-05-28 03:39:41.061] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 24s ] executions total: 4807 successful: 4801 failed: 6 disconnect: 6 [ 25s ] executions total: 5141 successful: 5135 failed: 6 disconnect: 6 [ 26s ] executions total: 5478 successful: 5472 failed: 6 disconnect: 6 [2025-05-28 03:39:46.462] Connection error occurred! [ 27s ] executions total: 5607 successful: 5600 failed: 7 disconnect: 7 [2025-05-28 03:39:46.462] Connection error occurred! [2025-05-28 03:39:47.466] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 28s ] executions total: 5608 successful: 5601 failed: 7 disconnect: 7 [ 29s ] executions total: 5980 successful: 5973 failed: 7 disconnect: 7 [ 30s ] executions total: 6329 successful: 6322 failed: 7 disconnect: 7 [2025-05-28 03:39:52.672] Connection error occurred! [ 31s ] executions total: 6408 successful: 6400 failed: 8 disconnect: 8 [2025-05-28 03:39:52.672] Connection error occurred! [2025-05-28 03:39:53.676] Connection successfully recovered! The connection was restored in 1004 milliseconds. [ 32s ] executions total: 6409 successful: 6401 failed: 8 disconnect: 8 [ 33s ] executions total: 6794 successful: 6786 failed: 8 disconnect: 8 [ 34s ] executions total: 7146 successful: 7138 failed: 8 disconnect: 8 [2025-05-28 03:39:59.060] Connection error occurred! [ 35s ] executions total: 7209 successful: 7200 failed: 9 disconnect: 9 [2025-05-28 03:39:59.060] Connection error occurred! [2025-05-28 03:40:00.068] Connection successfully recovered! The connection was restored in 1008 milliseconds. [ 36s ] executions total: 7210 successful: 7201 failed: 9 disconnect: 9 [ 37s ] executions total: 7517 successful: 7508 failed: 9 disconnect: 9 [ 60s ] executions total: 7692 successful: 7683 failed: 9 disconnect: 9 Test Result: Total Executions: 7692 Successful Executions: 7683 Failed Executions: 9 Disconnection Counts: 9 Connection Information: Database Type: redis Host: redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local Port: 6379 Database: Table: User: default Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds Key: DB_CLIENT_BATCH_DATA_COUNT: 7683 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-redis-tqnwum --namespace ns-rdwsc ` pod/test-db-client-executionloop-redis-tqnwum patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-redis-tqnwum" force deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo 'echo "set mykey \"satbh\"" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. OK add consistent data satbh Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo 'echo "set mykey \"satbh\"" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. READONLY You can't write against a read only replica. check add consistent data readonly Success LB_TYPE is set to: internet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose redis-tqnwum --auto-approve --force=true --type internet --enable true --components redis --role-selector primary --namespace ns-rdwsc ` OpsRequest redis-tqnwum-expose-ppdhw created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-expose-ppdhw -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-expose-ppdhw ns-rdwsc Expose redis-tqnwum redis Running 0/1 May 28,2025 11:40 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-expose-ppdhw ns-rdwsc Expose redis-tqnwum redis Succeed 1/1 May 28,2025 11:40 UTC+0800 check ops status done ops_status:redis-tqnwum-expose-ppdhw ns-rdwsc Expose redis-tqnwum redis Succeed 1/1 May 28,2025 11:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-expose-ppdhw --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-expose-ppdhw patched `kbcli cluster delete-ops --name redis-tqnwum-expose-ppdhw --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-expose-ppdhw deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file satbh Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-redis-tqnwum --namespace ns-rdwsc ` Error from server (NotFound): pods "test-db-client-connectionstress-redis-tqnwum" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-redis-tqnwum" not found `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-redis-tqnwum namespace: ns-rdwsc spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local" - "--user" - "default" - "--password" - "3L7022eJcr" - "--port" - "6379" - "--database" - "db0" - "--dbtype" - "redis" - "--test" - "connectionstress" - "--connections" - "5000" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-redis-tqnwum.yaml` pod/test-db-client-connectionstress-redis-tqnwum created apply test-db-client-connectionstress-redis-tqnwum.yaml Success `rm -rf test-db-client-connectionstress-redis-tqnwum.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 21s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 27s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 33s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 39s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 44s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 50s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 1/1 Running 0 62s check pod test-db-client-connectionstress-redis-tqnwum status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-redis-tqnwum 0/1 Completed 0 67s check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done --host redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local --user default --password 3L7022eJcr --port 6379 --database db0 --dbtype redis --test connectionstress --connections 5000 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] Test Result: Connection stress test results: Duration: 60 seconds Successful connections: 2779 Failed connections: 0 Connection Information: Database Type: redis Host: redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local Port: 6379 Database: db0 Table: User: default Org: Access Mode: mysql Test Type: connectionstress Connection Count: 5000 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-redis-tqnwum --namespace ns-rdwsc ` pod/test-db-client-connectionstress-redis-tqnwum patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-redis-tqnwum" force deleted check failover pod name failover pod name:redis-tqnwum-redis-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success cluster redis scale-out check cluster status before ops check cluster status done cluster_status:Running No resources found in redis-tqnwum namespace. `kbcli cluster scale-out redis-tqnwum --auto-approve --force=true --components redis --replicas 1 --namespace ns-rdwsc ` OpsRequest redis-tqnwum-horizontalscaling-gpk7w created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-horizontalscaling-gpk7w -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-horizontalscaling-gpk7w ns-rdwsc HorizontalScaling redis-tqnwum redis Running 0/1 May 28,2025 11:43 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-2 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:43 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 redis-tqnwum-redis-2 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done No resources found in redis-tqnwum namespace. check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-horizontalscaling-gpk7w ns-rdwsc HorizontalScaling redis-tqnwum redis Succeed 1/1 May 28,2025 11:43 UTC+0800 check ops status done ops_status:redis-tqnwum-horizontalscaling-gpk7w ns-rdwsc HorizontalScaling redis-tqnwum redis Succeed 1/1 May 28,2025 11:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-horizontalscaling-gpk7w --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-horizontalscaling-gpk7w patched `kbcli cluster delete-ops --name redis-tqnwum-horizontalscaling-gpk7w --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-horizontalscaling-gpk7w deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success get redis-sentinel replicas cluster redis scale-in check cluster status before ops check cluster status done cluster_status:Running No resources found in redis-tqnwum namespace. `kbcli cluster scale-in redis-tqnwum --auto-approve --force=true --components redis --replicas 1 --namespace ns-rdwsc ` OpsRequest redis-tqnwum-horizontalscaling-hvk5p created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-horizontalscaling-hvk5p -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-horizontalscaling-hvk5p ns-rdwsc HorizontalScaling redis-tqnwum redis Running 0/1 May 28,2025 11:44 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done No resources found in redis-tqnwum namespace. check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-horizontalscaling-hvk5p ns-rdwsc HorizontalScaling redis-tqnwum redis Succeed 1/1 May 28,2025 11:44 UTC+0800 check ops status done ops_status:redis-tqnwum-horizontalscaling-hvk5p ns-rdwsc HorizontalScaling redis-tqnwum redis Succeed 1/1 May 28,2025 11:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-horizontalscaling-hvk5p --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-horizontalscaling-hvk5p patched `kbcli cluster delete-ops --name redis-tqnwum-horizontalscaling-hvk5p --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-horizontalscaling-hvk5p deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success cluster does not need to check monitor currently check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-tqnwum" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 action: partition mode: all target: mode: all selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-redis-tqnwum.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-redis-tqnwum created apply test-chaos-mesh-networkpartition-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-networkpartition-redis-tqnwum.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-tqnwum" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-tqnwum" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-redis-tqnwum.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-redis-tqnwum created apply test-chaos-mesh-podfailure-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-podfailure-redis-tqnwum.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-tqnwum" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-1 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart redis-tqnwum --auto-approve --force=true --namespace ns-rdwsc ` OpsRequest redis-tqnwum-restart-9l8sh created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-restart-9l8sh -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 0/5 May 28,2025 11:51 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Terminating secondary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:36 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:37 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 pod_status:Init:0/3 pod_status:Init:0/3 pod_status:Init:0/3 pod_status:Init:2/3 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 3/5 May 28,2025 11:51 UTC+0800 ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 3/5 May 28,2025 11:51 UTC+0800 ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 3/5 May 28,2025 11:51 UTC+0800 ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 3/5 May 28,2025 11:51 UTC+0800 ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 4/5 May 28,2025 11:51 UTC+0800 ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 4/5 May 28,2025 11:51 UTC+0800 ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Running 4/5 May 28,2025 11:51 UTC+0800 check ops status done ops_status:redis-tqnwum-restart-9l8sh ns-rdwsc Restart redis-tqnwum redis,redis-sentinel Succeed 5/5 May 28,2025 11:51 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-restart-9l8sh --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-restart-9l8sh patched `kbcli cluster delete-ops --name redis-tqnwum-restart-9l8sh --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-restart-9l8sh deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale redis-tqnwum --auto-approve --force=true --components redis --cpu 200m --memory 0.6Gi --namespace ns-rdwsc ` OpsRequest redis-tqnwum-verticalscaling-9558w created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-verticalscaling-9558w -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-verticalscaling-9558w ns-rdwsc VerticalScaling redis-tqnwum redis Running 0/2 May 28,2025 11:54 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:55 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-verticalscaling-9558w ns-rdwsc VerticalScaling redis-tqnwum redis Succeed 2/2 May 28,2025 11:54 UTC+0800 check ops status done ops_status:redis-tqnwum-verticalscaling-9558w ns-rdwsc VerticalScaling redis-tqnwum redis Succeed 2/2 May 28,2025 11:54 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-verticalscaling-9558w --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-verticalscaling-9558w patched `kbcli cluster delete-ops --name redis-tqnwum-verticalscaling-9558w --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-verticalscaling-9558w deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-tqnwum" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-redis-tqnwum.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-redis-tqnwum created apply test-chaos-mesh-networkbandwidthover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-redis-tqnwum.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:55 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role No resources found in ns-rdwsc namespace. primary: redis-tqnwum-redis-0 redis-tqnwum-redis-1;secondary: No resources found in ns-rdwsc namespace. primary: redis-tqnwum-redis-0 redis-tqnwum-redis-1;secondary: No resources found in ns-rdwsc namespace. primary: redis-tqnwum-redis-0 redis-tqnwum-redis-1;secondary: check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-tqnwum" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-tqnwum" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-redis-tqnwum.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-redis-tqnwum created apply test-chaos-mesh-networkdelay-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-networkdelay-redis-tqnwum.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:55 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-tqnwum" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-tqnwum" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-redis-tqnwum.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-redis-tqnwum created apply test-chaos-mesh-networkduplicate-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-networkduplicate-redis-tqnwum.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:55 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-tqnwum" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "redis-tqnwum-redis-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done check failover pod name failover pod name:redis-tqnwum-redis-1 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file satbh Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file satbh Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover fullcpuover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum created apply test-chaos-mesh-fullcpuover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum created apply test-chaos-mesh-fullcpuover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found failover pod name:redis-tqnwum-redis-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum created apply test-chaos-mesh-fullcpuover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum patched stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" force deleted failover pod name:redis-tqnwum-redis-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum created apply test-chaos-mesh-fullcpuover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found failover pod name:redis-tqnwum-redis-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum created apply test-chaos-mesh-fullcpuover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found failover pod name:redis-tqnwum-redis-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpuover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpuover-redis-tqnwum created apply test-chaos-mesh-fullcpuover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-fullcpuover-redis-tqnwum.yaml` fullcpuover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:07 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:54 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:52 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 11:51 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 11:51 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpuover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpuover-redis-tqnwum" not found check failover pod name:redis-tqnwum-redis-0 failover fullcpuover Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop redis-tqnwum --auto-approve --force=true --namespace ns-rdwsc ` OpsRequest redis-tqnwum-stop-74g9q created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-stop-74g9q -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-stop-74g9q ns-rdwsc Stop redis-tqnwum redis,redis-sentinel Running 0/5 May 28,2025 12:22 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Stopped May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-stop-74g9q ns-rdwsc Stop redis-tqnwum redis,redis-sentinel Succeed 5/5 May 28,2025 12:22 UTC+0800 check ops status done ops_status:redis-tqnwum-stop-74g9q ns-rdwsc Stop redis-tqnwum redis,redis-sentinel Succeed 5/5 May 28,2025 12:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-stop-74g9q --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-stop-74g9q patched `kbcli cluster delete-ops --name redis-tqnwum-stop-74g9q --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-stop-74g9q deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start redis-tqnwum --force=true --namespace ns-rdwsc ` OpsRequest redis-tqnwum-start-556hp created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-start-556hp -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-start-556hp ns-rdwsc Start redis-tqnwum redis,redis-sentinel Running 0/5 May 28,2025 12:23 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-start-556hp ns-rdwsc Start redis-tqnwum redis,redis-sentinel Succeed 5/5 May 28,2025 12:23 UTC+0800 check ops status done ops_status:redis-tqnwum-start-556hp ns-rdwsc Start redis-tqnwum redis,redis-sentinel Succeed 5/5 May 28,2025 12:23 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-start-556hp --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-start-556hp patched `kbcli cluster delete-ops --name redis-tqnwum-start-556hp --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-start-556hp deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-redis-tqnwum created apply test-chaos-mesh-oom-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-oom-redis-tqnwum.yaml` check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-redis-tqnwum patched check failover pod name failover pod name:redis-tqnwum-redis-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-redis-tqnwum.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-redis-tqnwum created apply test-chaos-mesh-oom-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-oom-redis-tqnwum.yaml` check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-redis-tqnwum" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-redis-tqnwum patched check failover pod name:redis-tqnwum-redis-1 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-tqnwum" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-redis-tqnwum.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-redis-tqnwum created apply test-chaos-mesh-networklossover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-networklossover-redis-tqnwum.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role No resources found in ns-rdwsc namespace. primary: redis-tqnwum-redis-0 redis-tqnwum-redis-1;secondary: No resources found in ns-rdwsc namespace. primary: redis-tqnwum-redis-0 redis-tqnwum-redis-1;secondary: No resources found in ns-rdwsc namespace. primary: redis-tqnwum-redis-0 redis-tqnwum-redis-1;secondary: check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-tqnwum" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-tqnwum" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-redis-tqnwum.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-redis-tqnwum created apply test-chaos-mesh-timeoffset-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-timeoffset-redis-tqnwum.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-tqnwum" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod redis-tqnwum-redis-0 --namespace ns-rdwsc -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-8-216.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod redis-tqnwum-redis-1 --namespace ns-rdwsc -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-5-153.us-west-2.compute.internal success kubectl get pod redis-tqnwum-redis-sentinel-0 --namespace ns-rdwsc -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-8-216.us-west-2.compute.internal success Multiple pods on the same node cluster configure component_tmp: redis apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: redis-tqnwum-reconfiguring- namespace: ns-rdwsc spec: type: Reconfiguring clusterName: redis-tqnwum force: true reconfigures: - componentName: redis parameters: - key: maxclients value: '10001' check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_redis-tqnwum.yaml` opsrequest.operations.kubeblocks.io/redis-tqnwum-reconfiguring-vm5t4 created create test_ops_cluster_redis-tqnwum.yaml Success `rm -rf test_ops_cluster_redis-tqnwum.yaml` check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-reconfiguring-vm5t4 ns-rdwsc Reconfiguring redis-tqnwum redis,redis Running -/- May 28,2025 12:32 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:33 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-reconfiguring-vm5t4 ns-rdwsc Reconfiguring redis-tqnwum redis,redis Succeed -/- May 28,2025 12:32 UTC+0800 check ops status done ops_status:redis-tqnwum-reconfiguring-vm5t4 ns-rdwsc Reconfiguring redis-tqnwum redis,redis Succeed -/- May 28,2025 12:32 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-reconfiguring-vm5t4 --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-reconfiguring-vm5t4 patched `kbcli cluster delete-ops --name redis-tqnwum-reconfiguring-vm5t4 --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-reconfiguring-vm5t4 deleted component_config:redis check config variables `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. current value_actual: 10001 configure:[maxclients] result actual:[10001] equal expected:[10001] `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file satbh Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success 9 test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-tqnwum" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-redis-tqnwum.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-redis-tqnwum created apply test-chaos-mesh-networkcorruptover-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-redis-tqnwum.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:33 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-0;secondary: redis-tqnwum-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-tqnwum" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-0 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success get redis-sentinel replicas test failover delete pod:redis-tqnwum-redis-0 `kubectl delete pod redis-tqnwum-redis-0 --namespace ns-rdwsc ` pod "redis-tqnwum-redis-0" deleted check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:37 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done check failover pod name failover pod name:redis-tqnwum-redis-1 failover Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: benchtest-redis-tqnwum namespace: ns-rdwsc spec: containers: - name: test-benchmark imagePullPolicy: IfNotPresent image: docker.io/apecloud/redis-benchmark:latest args: - "-h" - "redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local" - "-p" - "6379" - "-a" - "3L7022eJcr" - "-n" - "1000" - "-c" - "2" - "-q" restartPolicy: Never `kubectl apply -f benchtest-redis-tqnwum.yaml` pod/benchtest-redis-tqnwum created apply benchtest-redis-tqnwum.yaml Success `rm -rf benchtest-redis-tqnwum.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 1/1 Running 0 12s check pod benchtest-redis-tqnwum status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 0/1 Completed 0 17s `kubectl logs benchtest-redis-tqnwum --tail 30 --namespace ns-rdwsc ` PING_INLINE: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) PING_INLINE: rps=3091.6 (overall: 3091.6) avg_msec=0.601 (overall: 0.601) PING_INLINE: 3067.48 requests per second, p50=0.383 msec PING_MBULK: rps=1988.0 (overall: 2923.5) avg_msec=0.667 (overall: 0.667) PING_MBULK: 2481.39 requests per second, p50=0.375 msec SET: rps=51.8 (overall: 1000.0) avg_msec=0.581 (overall: 0.581) SET: rps=1788.8 (overall: 1750.0) avg_msec=1.144 (overall: 1.128) SET: rps=1916.3 (overall: 1831.1) avg_msec=0.986 (overall: 1.055) SET: 1785.71 requests per second, p50=0.487 msec GET: rps=1649.4 (overall: 2059.7) avg_msec=0.943 (overall: 0.943) GET: 2538.07 requests per second, p50=0.383 msec INCR: rps=502.0 (overall: 2680.9) avg_msec=0.425 (overall: 0.425) INCR: rps=2223.1 (overall: 2295.3) avg_msec=0.954 (overall: 0.857) INCR: 2079.00 requests per second, p50=0.399 msec LPUSH: rps=541.8 (overall: 2125.0) avg_msec=0.899 (overall: 0.899) LPUSH: rps=1920.0 (overall: 1961.8) avg_msec=1.040 (overall: 1.009) LPUSH: rps=924.3 (overall: 1500.9) avg_msec=1.982 (overall: 1.275) LPUSH: 1594.90 requests per second, p50=0.407 msec RPUSH: rps=1636.0 (overall: 2247.3) avg_msec=0.879 (overall: 0.879) RPUSH: rps=1865.1 (overall: 2025.3) avg_msec=0.930 (overall: 0.906) RPUSH: 2053.39 requests per second, p50=0.431 msec LPOP: rps=1372.0 (overall: 1795.8) avg_msec=1.101 (overall: 1.101) LPOP: rps=1525.9 (overall: 1642.5) avg_msec=1.140 (overall: 1.121) LPOP: 1730.10 requests per second, p50=0.455 msec RPOP: rps=1268.0 (overall: 2881.8) avg_msec=0.680 (overall: 0.680) RPOP: rps=1486.1 (overall: 1911.4) avg_msec=1.155 (overall: 0.937) RPOP: 2004.01 requests per second, p50=0.415 msec SADD: rps=1442.2 (overall: 3351.9) avg_msec=0.575 (overall: 0.575) SADD: 3322.26 requests per second, p50=0.351 msec HSET: rps=414.3 (overall: 1962.3) avg_msec=0.411 (overall: 0.411) HSET: rps=2569.7 (overall: 2463.8) avg_msec=0.852 (overall: 0.791) HSET: 2096.44 requests per second, p50=0.375 msec SPOP: rps=856.6 (overall: 2945.2) avg_msec=0.354 (overall: 0.354) SPOP: rps=2513.9 (overall: 2611.1) avg_msec=0.861 (overall: 0.732) SPOP: 2770.08 requests per second, p50=0.399 msec ZADD: rps=2196.0 (overall: 2652.2) avg_msec=0.649 (overall: 0.649) ZADD: 2777.78 requests per second, p50=0.415 msec ZPOPMIN: rps=872.5 (overall: 2354.8) avg_msec=0.833 (overall: 0.833) ZPOPMIN: rps=2864.5 (overall: 2726.7) avg_msec=0.590 (overall: 0.647) ZPOPMIN: 2762.43 requests per second, p50=0.367 msec LPUSH (needed to benchmark LRANGE): rps=1868.0 (overall: 2057.3) avg_msec=0.960 (overall: 0.960) LPUSH (needed to benchmark LRANGE): rps=1186.5 (overall: 1599.2) avg_msec=1.208 (overall: 1.057) LPUSH (needed to benchmark LRANGE): 1680.67 requests per second, p50=0.487 msec LRANGE_100 (first 100 elements): rps=1500.0 (overall: 2976.2) avg_msec=0.638 (overall: 0.638) LRANGE_100 (first 100 elements): 2967.36 requests per second, p50=0.391 msec LRANGE_300 (first 300 elements): rps=43.8 (overall: 366.7) avg_msec=0.584 (overall: 0.584) LRANGE_300 (first 300 elements): rps=1337.3 (overall: 1234.0) avg_msec=1.530 (overall: 1.500) LRANGE_300 (first 300 elements): rps=666.7 (overall: 966.3) avg_msec=2.901 (overall: 1.956) LRANGE_300 (first 300 elements): 1351.35 requests per second, p50=0.463 msec LRANGE_500 (first 500 elements): rps=424.0 (overall: 2650.0) avg_msec=0.611 (overall: 0.611) LRANGE_500 (first 500 elements): rps=1382.5 (overall: 1556.7) avg_msec=1.271 (overall: 1.117) LRANGE_500 (first 500 elements): rps=2012.0 (overall: 1767.5) avg_msec=0.838 (overall: 0.970) LRANGE_500 (first 500 elements): 1694.92 requests per second, p50=0.535 msec LRANGE_600 (first 600 elements): rps=1160.0 (overall: 1464.6) avg_msec=1.197 (overall: 1.197) LRANGE_600 (first 600 elements): rps=1944.9 (overall: 1734.5) avg_msec=0.840 (overall: 0.972) LRANGE_600 (first 600 elements): 1612.90 requests per second, p50=0.551 msec MSET (10 keys): rps=601.6 (overall: 1935.9) avg_msec=1.012 (overall: 1.012) MSET (10 keys): rps=1924.3 (overall: 1927.1) avg_msec=1.012 (overall: 1.012) MSET (10 keys): 1956.95 requests per second, p50=0.439 msec XADD: rps=143.4 (overall: 553.8) avg_msec=3.585 (overall: 3.585) XADD: rps=2003.9 (overall: 1708.5) avg_msec=0.891 (overall: 1.069) XADD: rps=1672.0 (overall: 1692.4) avg_msec=1.303 (overall: 1.170) XADD: 1730.10 requests per second, p50=0.415 msec `kubectl delete pod benchtest-redis-tqnwum --force --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-redis-tqnwum" force deleted LB_TYPE is set to: internet No resources found in ns-rdwsc namespace. `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 apiVersion: v1 kind: Pod metadata: name: benchtest-redis-tqnwum namespace: ns-rdwsc spec: containers: - name: test-benchmark imagePullPolicy: IfNotPresent image: docker.io/apecloud/redis-benchmark:latest args: - "-h" - "a5929afe52eb34193a0d799ac18ee694-f19758b6ef84745a.elb.us-west-2.amazonaws.com" - "-p" - "6379" - "-a" - "3L7022eJcr" - "-n" - "1000" - "-c" - "2" - "-q" restartPolicy: Never `kubectl apply -f benchtest-redis-tqnwum.yaml` pod/benchtest-redis-tqnwum created apply benchtest-redis-tqnwum.yaml Success `rm -rf benchtest-redis-tqnwum.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 1/1 Running 0 11s check pod benchtest-redis-tqnwum status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-redis-tqnwum 0/1 Completed 0 17s `kubectl logs benchtest-redis-tqnwum --tail 30 --namespace ns-rdwsc ` PING_INLINE: rps=0.0 (overall: -nan) avg_msec=-nan (overall: -nan) PING_INLINE: rps=2681.3 (overall: 2681.3) avg_msec=0.723 (overall: 0.723) PING_INLINE: 2309.47 requests per second, p50=0.519 msec PING_MBULK: rps=476.0 (overall: 1888.9) avg_msec=1.023 (overall: 1.023) PING_MBULK: rps=2768.9 (overall: 2592.4) avg_msec=0.701 (overall: 0.748) PING_MBULK: 2444.99 requests per second, p50=0.519 msec SET: rps=1031.9 (overall: 1715.2) avg_msec=1.109 (overall: 1.109) SET: rps=1506.0 (overall: 1584.6) avg_msec=1.329 (overall: 1.240) SET: 1709.40 requests per second, p50=0.551 msec GET: rps=392.0 (overall: 1606.6) avg_msec=1.226 (overall: 1.226) GET: rps=2191.2 (overall: 2076.9) avg_msec=0.849 (overall: 0.906) GET: 1893.94 requests per second, p50=0.495 msec INCR: rps=199.2 (overall: 1666.7) avg_msec=0.688 (overall: 0.688) INCR: rps=1318.7 (overall: 1355.9) avg_msec=1.574 (overall: 1.458) INCR: rps=1282.9 (overall: 1321.4) avg_msec=1.530 (overall: 1.491) INCR: 1468.43 requests per second, p50=0.575 msec LPUSH: rps=486.1 (overall: 1326.1) avg_msec=1.447 (overall: 1.447) LPUSH: rps=1513.9 (overall: 1463.6) avg_msec=1.303 (overall: 1.338) LPUSH: rps=1724.0 (overall: 1573.4) avg_msec=1.108 (overall: 1.232) LPUSH: 1360.54 requests per second, p50=0.583 msec RPUSH: rps=876.5 (overall: 2115.4) avg_msec=0.930 (overall: 0.930) RPUSH: rps=1537.8 (overall: 1707.0) avg_msec=1.237 (overall: 1.125) RPUSH: 1677.85 requests per second, p50=0.551 msec LPOP: rps=16.0 (overall: 2000.0) avg_msec=0.636 (overall: 0.636) LPOP: rps=1928.3 (overall: 1928.9) avg_msec=1.013 (overall: 1.010) LPOP: rps=1569.7 (overall: 1750.0) avg_msec=1.250 (overall: 1.117) LPOP: 1805.05 requests per second, p50=0.543 msec RPOP: rps=892.4 (overall: 1142.9) avg_msec=1.533 (overall: 1.533) RPOP: rps=1187.3 (overall: 1167.8) avg_msec=1.645 (overall: 1.597) RPOP: 1447.18 requests per second, p50=0.575 msec SADD: rps=0.0 (overall: 0.0) avg_msec=-nan (overall: -nan) SADD: rps=2208.0 (overall: 2199.2) avg_msec=0.891 (overall: 0.891) SADD: 2475.25 requests per second, p50=0.487 msec HSET: rps=637.5 (overall: 1720.4) avg_msec=1.129 (overall: 1.129) HSET: rps=1260.0 (overall: 1384.8) avg_msec=1.569 (overall: 1.421) HSET: rps=784.9 (overall: 1131.3) avg_msec=2.464 (overall: 1.727) HSET: 1285.35 requests per second, p50=0.599 msec SPOP: rps=628.0 (overall: 2616.7) avg_msec=0.737 (overall: 0.737) SPOP: rps=1628.8 (overall: 1811.7) avg_msec=0.925 (overall: 0.875) SPOP: 1795.33 requests per second, p50=0.543 msec ZADD: rps=138.9 (overall: 2692.3) avg_msec=0.696 (overall: 0.696) ZADD: rps=2330.7 (overall: 2348.5) avg_msec=0.830 (overall: 0.822) ZADD: rps=1346.6 (overall: 1860.2) avg_msec=1.496 (overall: 1.060) ZADD: 1890.36 requests per second, p50=0.575 msec ZPOPMIN: rps=2852.0 (overall: 3113.5) avg_msec=0.611 (overall: 0.611) ZPOPMIN: 2824.86 requests per second, p50=0.487 msec LPUSH (needed to benchmark LRANGE): rps=924.3 (overall: 1917.4) avg_msec=1.011 (overall: 1.011) LPUSH (needed to benchmark LRANGE): rps=1964.1 (overall: 1948.9) avg_msec=0.986 (overall: 0.994) LPUSH (needed to benchmark LRANGE): 1976.28 requests per second, p50=0.543 msec LRANGE_100 (first 100 elements): rps=916.0 (overall: 2081.8) avg_msec=0.936 (overall: 0.936) LRANGE_100 (first 100 elements): rps=1856.6 (overall: 1925.2) avg_msec=0.889 (overall: 0.904) LRANGE_100 (first 100 elements): 2000.00 requests per second, p50=0.551 msec LRANGE_300 (first 300 elements): rps=824.7 (overall: 1952.8) avg_msec=0.934 (overall: 0.934) LRANGE_300 (first 300 elements): rps=1318.7 (overall: 1507.0) avg_msec=1.389 (overall: 1.214) LRANGE_300 (first 300 elements): rps=1573.7 (overall: 1534.5) avg_msec=1.192 (overall: 1.205) LRANGE_300 (first 300 elements): 1572.33 requests per second, p50=0.631 msec LRANGE_500 (first 500 elements): rps=1298.8 (overall: 1495.4) avg_msec=0.962 (overall: 0.962) LRANGE_500 (first 500 elements): rps=1557.8 (overall: 1528.8) avg_msec=1.304 (overall: 1.149) LRANGE_500 (first 500 elements): rps=1088.0 (overall: 1375.5) avg_msec=1.671 (overall: 1.292) LRANGE_500 (first 500 elements): 1377.41 requests per second, p50=0.663 msec LRANGE_600 (first 600 elements): rps=1322.7 (overall: 1395.0) avg_msec=1.254 (overall: 1.254) LRANGE_600 (first 600 elements): rps=1142.9 (overall: 1265.3) avg_msec=1.436 (overall: 1.339) LRANGE_600 (first 600 elements): rps=912.0 (overall: 1145.9) avg_msec=2.188 (overall: 1.567) LRANGE_600 (first 600 elements): 1194.74 requests per second, p50=0.767 msec MSET (10 keys): rps=394.4 (overall: 664.4) avg_msec=2.958 (overall: 2.958) MSET (10 keys): rps=1231.1 (overall: 1020.0) avg_msec=1.594 (overall: 1.925) MSET (10 keys): rps=645.4 (overall: 875.6) avg_msec=2.133 (overall: 1.984) MSET (10 keys): rps=1653.4 (overall: 1092.0) avg_msec=1.579 (overall: 1.813) MSET (10 keys): 1044.93 requests per second, p50=0.615 msec XADD: rps=1151.4 (overall: 1513.1) avg_msec=1.137 (overall: 1.137) XADD: rps=1612.0 (overall: 1569.2) avg_msec=1.343 (overall: 1.257) XADD: 1483.68 requests per second, p50=0.575 msec `kubectl delete pod benchtest-redis-tqnwum --force --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-redis-tqnwum" force deleted `kubectl get pvc -l app.kubernetes.io/instance=redis-tqnwum,apps.kubeblocks.io/component-name=redis,apps.kubeblocks.io/vct-name=data --namespace ns-rdwsc ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in redis-tqnwum namespace. `kbcli cluster volume-expand redis-tqnwum --auto-approve --force=true --components redis --volume-claim-templates data --storage 6Gi --namespace ns-rdwsc ` OpsRequest redis-tqnwum-volumeexpansion-fj2rm created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-volumeexpansion-fj2rm -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-volumeexpansion-fj2rm ns-rdwsc VolumeExpansion redis-tqnwum redis Running 0/2 May 28,2025 12:40 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:37 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done No resources found in redis-tqnwum namespace. check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-volumeexpansion-fj2rm ns-rdwsc VolumeExpansion redis-tqnwum redis Succeed 2/2 May 28,2025 12:40 UTC+0800 check ops status done ops_status:redis-tqnwum-volumeexpansion-fj2rm ns-rdwsc VolumeExpansion redis-tqnwum redis Succeed 2/2 May 28,2025 12:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-volumeexpansion-fj2rm --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-volumeexpansion-fj2rm patched `kbcli cluster delete-ops --name redis-tqnwum-volumeexpansion-fj2rm --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-volumeexpansion-fj2rm deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-tqnwum" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-redis-tqnwum.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-redis-tqnwum created apply test-chaos-mesh-dnsrandom-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-dnsrandom-redis-tqnwum.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:37 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-redis-tqnwum" force deleted dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-redis-tqnwum patched check failover pod name failover pod name:redis-tqnwum-redis-1 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-tqnwum" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-tqnwum" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-redis-tqnwum namespace: ns-rdwsc spec: selector: namespaces: - ns-rdwsc labelSelectors: apps.kubeblocks.io/pod-name: redis-tqnwum-redis-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-redis-tqnwum.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-redis-tqnwum created apply test-chaos-mesh-dnserror-redis-tqnwum.yaml Success `rm -rf test-chaos-mesh-dnserror-redis-tqnwum.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:37 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-redis-tqnwum --namespace ns-rdwsc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-tqnwum" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-redis-tqnwum" not found check failover pod name failover pod name:redis-tqnwum-redis-1 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-0 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update redis-tqnwum --termination-policy=WipeOut --namespace ns-rdwsc ` cluster.apps.kubeblocks.io/redis-tqnwum updated (no change) check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-0 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:37 UTC+0800 redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-0 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl get backupschedule -l app.kubernetes.io/instance=redis-tqnwum ` `kubectl get backupschedule redis-tqnwum-redis-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched check backup status `kbcli cluster list-backups redis-tqnwum --namespace ns-rdwsc ` No backups found in ns-rdwsc namespace. No backups found in ns-rdwsc namespace. backup_status:-- No backups found in ns-rdwsc namespace. backup_status:-- No backups found in ns-rdwsc namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=redis-tqnwum ` `kubectl get backupschedule redis-tqnwum-redis-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backup_status:redis-tqnwum-datafile-Running backup_status:redis-tqnwum-datafile-Running check backup status done backup_status:redis-tqnwum-datafile-20250528044901 ns-rdwsc redis-tqnwum datafile Completed 22173 16s Delete May 28,2025 12:49 UTC+0800 May 28,2025 12:49 UTC+0800 Jun 04,2025 12:49 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=redis-tqnwum ` `kubectl get backupschedule redis-tqnwum-redis-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/redis-tqnwum-redis-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "redis-tqnwum-backup" not found `kbcli cluster describe-backup --names redis-tqnwum-datafile-20250528044901 --namespace ns-rdwsc ` Name: redis-tqnwum-datafile-20250528044901 Cluster: redis-tqnwum Namespace: ns-rdwsc Spec: Method: datafile Policy Name: redis-tqnwum-redis-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-redis-tqnwum-datafile-20250528044901-ecc94324 TargetPodName: redis-tqnwum-redis-0 Phase: Completed Start Time: May 28,2025 12:49 UTC+0800 Completion Time: May 28,2025 12:49 UTC+0800 Status: Phase: Completed Total Size: 22173 ActionSet Name: redis-physical-br Repository: backuprepo-kbcli-test Duration: 16s Expiration Time: Jun 04,2025 12:49 UTC+0800 Start Time: May 28,2025 12:49 UTC+0800 Completion Time: May 28,2025 12:49 UTC+0800 Path: /ns-rdwsc/redis-tqnwum-f95ab685-ceb2-4afd-a13a-02a164b03976/redis/redis-tqnwum-datafile-20250528044901 Warning Events: `kbcli cluster restore redis-tqnwum-backup --backup redis-tqnwum-datafile-20250528044901 --namespace ns-rdwsc ` Cluster redis-tqnwum-backup created check cluster status `kbcli cluster list redis-tqnwum-backup --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum-backup ns-rdwsc redis WipeOut Creating May 28,2025 12:49 UTC+0800 clusterdefinition.kubeblocks.io/name=redis cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum-backup --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-backup-redis-0 ns-rdwsc redis-tqnwum-backup redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:50 UTC+0800 redis-tqnwum-backup-redis-1 ns-rdwsc redis-tqnwum-backup redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:50 UTC+0800 redis-tqnwum-backup-redis-sentinel-0 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:49 UTC+0800 redis-tqnwum-backup-redis-sentinel-1 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:49 UTC+0800 redis-tqnwum-backup-redis-sentinel-2 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:49 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-backup-redis-0;secondary: redis-tqnwum-backup-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum-backup` set secret: redis-tqnwum-backup-redis-account-default `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-backup-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-backup-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kbcli cluster describe-backup --names redis-tqnwum-datafile-20250528044901 --namespace ns-rdwsc ` Name: redis-tqnwum-datafile-20250528044901 Cluster: redis-tqnwum Namespace: ns-rdwsc Spec: Method: datafile Policy Name: redis-tqnwum-redis-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-redis-tqnwum-datafile-20250528044901-ecc94324 TargetPodName: redis-tqnwum-redis-0 Phase: Completed Start Time: May 28,2025 12:49 UTC+0800 Completion Time: May 28,2025 12:49 UTC+0800 Status: Phase: Completed Total Size: 22173 ActionSet Name: redis-physical-br Repository: backuprepo-kbcli-test Duration: 16s Expiration Time: Jun 04,2025 12:49 UTC+0800 Start Time: May 28,2025 12:49 UTC+0800 Completion Time: May 28,2025 12:49 UTC+0800 Path: /ns-rdwsc/redis-tqnwum-f95ab685-ceb2-4afd-a13a-02a164b03976/redis/redis-tqnwum-datafile-20250528044901 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum-backup` set secret: redis-tqnwum-backup-redis-account-default `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.2.4 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:3a23a5ac67fe7008 redis_mode:standalone os:Linux 5.10.236-228.935.amzn2.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:21ca7afbfe91547e90f252c7b743305964768c32 tcp_port:6379 server_time_usec:1748407900425272 uptime_in_seconds:46 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:3577436 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo " echo \"CONFIG GET maxclients\" | redis-cli -h redis-tqnwum-backup-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-backup-redis-0 --namespace ns-rdwsc -- bash ` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 delete cluster redis-tqnwum-backup `kbcli cluster delete redis-tqnwum-backup --auto-approve --namespace ns-rdwsc ` Cluster redis-tqnwum-backup deleted pod_info:redis-tqnwum-backup-redis-0 3/3 Running 0 67s redis-tqnwum-backup-redis-1 3/3 Running 0 49s redis-tqnwum-backup-redis-sentinel-0 2/2 Terminating 0 2m10s redis-tqnwum-backup-redis-sentinel-1 2/2 Terminating 0 2m10s redis-tqnwum-backup-redis-sentinel-2 2/2 Terminating 0 2m10s No resources found in ns-rdwsc namespace. delete cluster pod done No resources found in ns-rdwsc namespace. check cluster resource non-exist OK: pvc No resources found in ns-rdwsc namespace. delete cluster done No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups redis-tqnwum-datafile-20250528044901 --namespace ns-rdwsc ` backup.dataprotection.kubeblocks.io/redis-tqnwum-datafile-20250528044901 patched `kbcli cluster delete-backup redis-tqnwum --name redis-tqnwum-datafile-20250528044901 --force --auto-approve --namespace ns-rdwsc ` Backup redis-tqnwum-datafile-20250528044901 deleted No opsrequests found in ns-rdwsc namespace. cluster datafile backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-mb4vp -n kb-tgili -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-tgili get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-tgili -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-tgili.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-tgili.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5f4dfb568b-4g59b --namespace kb-tgili -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup redis-tqnwum --method datafile --namespace ns-rdwsc ` Backup backup-ns-rdwsc-redis-tqnwum-20250528125222 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-rdwsc-redis-tqnwum-20250528125222 -n ns-rdwsc check backup status `kbcli cluster list-backups redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-rdwsc-redis-tqnwum-20250528125222 ns-rdwsc redis-tqnwum datafile Running Delete May 28,2025 12:52 UTC+0800 backup_status:redis-tqnwum-datafile-Running backup_status:redis-tqnwum-datafile-Running check backup status done backup_status:backup-ns-rdwsc-redis-tqnwum-20250528125222 ns-rdwsc redis-tqnwum datafile Completed 22221 11s Delete May 28,2025 12:52 UTC+0800 May 28,2025 12:52 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "redis-tqnwum-backup" not found `kbcli cluster describe-backup --names backup-ns-rdwsc-redis-tqnwum-20250528125222 --namespace ns-rdwsc ` Name: backup-ns-rdwsc-redis-tqnwum-20250528125222 Cluster: redis-tqnwum Namespace: ns-rdwsc Spec: Method: datafile Policy Name: redis-tqnwum-redis-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-rdwsc-redis-tqnwum-20250528125222-6dc8fbb TargetPodName: redis-tqnwum-redis-0 Phase: Completed Start Time: May 28,2025 12:52 UTC+0800 Completion Time: May 28,2025 12:52 UTC+0800 Status: Phase: Completed Total Size: 22221 ActionSet Name: redis-physical-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: May 28,2025 12:52 UTC+0800 Completion Time: May 28,2025 12:52 UTC+0800 Path: /ns-rdwsc/redis-tqnwum-f95ab685-ceb2-4afd-a13a-02a164b03976/redis/backup-ns-rdwsc-redis-tqnwum-20250528125222 Warning Events: `kbcli cluster restore redis-tqnwum-backup --backup backup-ns-rdwsc-redis-tqnwum-20250528125222 --namespace ns-rdwsc ` Cluster redis-tqnwum-backup created check cluster status `kbcli cluster list redis-tqnwum-backup --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum-backup ns-rdwsc redis WipeOut Creating May 28,2025 12:52 UTC+0800 clusterdefinition.kubeblocks.io/name=redis cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum-backup --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-backup-redis-0 ns-rdwsc redis-tqnwum-backup redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:53 UTC+0800 redis-tqnwum-backup-redis-1 ns-rdwsc redis-tqnwum-backup redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:54 UTC+0800 redis-tqnwum-backup-redis-sentinel-0 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:52 UTC+0800 redis-tqnwum-backup-redis-sentinel-1 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:52 UTC+0800 redis-tqnwum-backup-redis-sentinel-2 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:52 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-backup-redis-0;secondary: redis-tqnwum-backup-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum-backup` set secret: redis-tqnwum-backup-redis-account-default `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-backup-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-backup-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-rdwsc-redis-tqnwum-20250528125222 --namespace ns-rdwsc ` Name: backup-ns-rdwsc-redis-tqnwum-20250528125222 Cluster: redis-tqnwum Namespace: ns-rdwsc Spec: Method: datafile Policy Name: redis-tqnwum-redis-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-rdwsc-redis-tqnwum-20250528125222-6dc8fbb TargetPodName: redis-tqnwum-redis-0 Phase: Completed Start Time: May 28,2025 12:52 UTC+0800 Completion Time: May 28,2025 12:52 UTC+0800 Status: Phase: Completed Total Size: 22221 ActionSet Name: redis-physical-br Repository: backuprepo-kbcli-test Duration: 11s Start Time: May 28,2025 12:52 UTC+0800 Completion Time: May 28,2025 12:52 UTC+0800 Path: /ns-rdwsc/redis-tqnwum-f95ab685-ceb2-4afd-a13a-02a164b03976/redis/backup-ns-rdwsc-redis-tqnwum-20250528125222 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum-backup` set secret: redis-tqnwum-backup-redis-account-default `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.2.4 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:3a23a5ac67fe7008 redis_mode:standalone os:Linux 5.10.236-228.935.amzn2.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:557427b53944df0a8a2061a4a6562f660477374f tcp_port:6379 server_time_usec:1748408086486380 uptime_in_seconds:49 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:3577622 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo " echo \"CONFIG GET maxclients\" | redis-cli -h redis-tqnwum-backup-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-backup-redis-0 --namespace ns-rdwsc -- bash ` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 delete cluster redis-tqnwum-backup `kbcli cluster delete redis-tqnwum-backup --auto-approve --namespace ns-rdwsc ` Cluster redis-tqnwum-backup deleted pod_info:redis-tqnwum-backup-redis-0 3/3 Running 0 71s redis-tqnwum-backup-redis-1 3/3 Running 0 49s redis-tqnwum-backup-redis-sentinel-0 2/2 Running 0 2m10s redis-tqnwum-backup-redis-sentinel-1 2/2 Running 0 2m10s redis-tqnwum-backup-redis-sentinel-2 2/2 Running 0 2m10s No resources found in ns-rdwsc namespace. delete cluster pod done No resources found in ns-rdwsc namespace. check cluster resource non-exist OK: pvc No resources found in ns-rdwsc namespace. delete cluster done No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: redis-tqnwum-rebuildinstance- namespace: ns-rdwsc spec: type: RebuildInstance clusterName: redis-tqnwum force: true rebuildFrom: - componentName: redis instances: - name: redis-tqnwum-redis-0 backupName: backup-ns-rdwsc-redis-tqnwum-20250528125222 inPlace: false check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_redis-tqnwum.yaml` opsrequest.operations.kubeblocks.io/redis-tqnwum-rebuildinstance-ng5g5 created create test_ops_cluster_redis-tqnwum.yaml Success `rm -rf test_ops_cluster_redis-tqnwum.yaml` check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-rebuildinstance-ng5g5 ns-rdwsc RebuildInstance redis-tqnwum redis Running 0/1 May 28,2025 12:55 UTC+0800 ops_status:redis-tqnwum-rebuildinstance-ng5g5 ns-rdwsc RebuildInstance redis-tqnwum redis Running 0/1 May 28,2025 12:55 UTC+0800 ops_status:redis-tqnwum-rebuildinstance-ng5g5 ns-rdwsc RebuildInstance redis-tqnwum redis Running 0/1 May 28,2025 12:55 UTC+0800 ops_status:redis-tqnwum-rebuildinstance-ng5g5 ns-rdwsc RebuildInstance redis-tqnwum redis Running 0/1 May 28,2025 12:55 UTC+0800 ops_status:redis-tqnwum-rebuildinstance-ng5g5 ns-rdwsc RebuildInstance redis-tqnwum redis Running 0/1 May 28,2025 12:55 UTC+0800 check ops status done ops_status:redis-tqnwum-rebuildinstance-ng5g5 ns-rdwsc RebuildInstance redis-tqnwum redis Succeed 1/1 May 28,2025 12:55 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-rebuildinstance-ng5g5 --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-rebuildinstance-ng5g5 patched `kbcli cluster delete-ops --name redis-tqnwum-rebuildinstance-ng5g5 --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-rebuildinstance-ng5g5 deleted check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-2 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:55 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-2 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-2 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-2 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-rdwsc-redis-tqnwum-20250528125222 --namespace ns-rdwsc ` backup.dataprotection.kubeblocks.io/backup-ns-rdwsc-redis-tqnwum-20250528125222 patched `kbcli cluster delete-backup redis-tqnwum --name backup-ns-rdwsc-redis-tqnwum-20250528125222 --force --auto-approve --namespace ns-rdwsc ` Backup backup-ns-rdwsc-redis-tqnwum-20250528125222 deleted No opsrequests found in ns-rdwsc namespace. cluster volume-snapshot backup `kbcli cluster backup redis-tqnwum --method volume-snapshot --namespace ns-rdwsc ` Backup backup-ns-rdwsc-redis-tqnwum-20250528125632 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-rdwsc-redis-tqnwum-20250528125632 -n ns-rdwsc check backup status `kbcli cluster list-backups redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-rdwsc-redis-tqnwum-20250528125632 ns-rdwsc redis-tqnwum volume-snapshot Running Delete May 28,2025 12:56 UTC+0800 backup_status:redis-tqnwum-volume-snapshot-Running backup_status:redis-tqnwum-volume-snapshot-Running backup_status:redis-tqnwum-volume-snapshot-Running backup_status:redis-tqnwum-volume-snapshot-Running backup_status:redis-tqnwum-volume-snapshot-Running backup_status:redis-tqnwum-volume-snapshot-Running check backup status done backup_status:backup-ns-rdwsc-redis-tqnwum-20250528125632 ns-rdwsc redis-tqnwum volume-snapshot Completed 6Gi 36s Delete May 28,2025 12:56 UTC+0800 May 28,2025 12:57 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "redis-tqnwum-backup" not found `kbcli cluster describe-backup --names backup-ns-rdwsc-redis-tqnwum-20250528125632 --namespace ns-rdwsc ` Name: backup-ns-rdwsc-redis-tqnwum-20250528125632 Cluster: redis-tqnwum Namespace: ns-rdwsc Spec: Method: volume-snapshot Policy Name: redis-tqnwum-redis-backup-policy Actions: createVolumeSnapshot-0: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2b1becf] goroutine 1 [running]: github.com/apecloud/kbcli/pkg/cmd/dataprotection.PrintBackupObjDescribe(0xc000c33a40, 0xc00096a588) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:480 +0x4cf github.com/apecloud/kbcli/pkg/cmd/dataprotection.DescribeBackups(0xc000c33a40, ***0xc00117fbb0?, 0x18fd69b?, 0xc001158908?***) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:458 +0x125 github.com/apecloud/kbcli/pkg/cmd/cluster.describeBackups(0x0?, ***0xc000642b00?, 0x0?, 0x19bba82000000000?***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:204 +0x66 github.com/apecloud/kbcli/pkg/cmd/cluster.NewDescribeBackupCmd.func1(0xc001120908?, ***0xc000642b00, 0x0, 0x4***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:195 +0xe5 github.com/spf13/cobra.(*Command).execute(0xc001120908, ***0xc000642ac0, 0x4, 0x4***) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xab1 github.com/spf13/cobra.(*Command).ExecuteC(0xc0008cc308) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 k8s.io/component-base/cli.run(0xc0008cc308) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:146 +0x290 k8s.io/component-base/cli.RunNoErrOutput(...) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:84 main.main() /home/runner/work/kbcli/kbcli/cmd/cli/main.go:31 +0x18 `kbcli cluster restore redis-tqnwum-backup --backup backup-ns-rdwsc-redis-tqnwum-20250528125632 --namespace ns-rdwsc ` Cluster redis-tqnwum-backup created check cluster status `kbcli cluster list redis-tqnwum-backup --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum-backup ns-rdwsc redis WipeOut Creating May 28,2025 12:57 UTC+0800 clusterdefinition.kubeblocks.io/name=redis cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum-backup --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-backup-redis-0 ns-rdwsc redis-tqnwum-backup redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:57 UTC+0800 redis-tqnwum-backup-redis-1 ns-rdwsc redis-tqnwum-backup redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:58 UTC+0800 redis-tqnwum-backup-redis-sentinel-0 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:57 UTC+0800 redis-tqnwum-backup-redis-sentinel-1 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:57 UTC+0800 redis-tqnwum-backup-redis-sentinel-2 ns-rdwsc redis-tqnwum-backup redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:57 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-backup-redis-0;secondary: redis-tqnwum-backup-redis-1 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum-backup` set secret: redis-tqnwum-backup-redis-account-default `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-backup-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-backup-redis-0 --namespace ns-rdwsc -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-rdwsc-redis-tqnwum-20250528125632 --namespace ns-rdwsc ` Name: backup-ns-rdwsc-redis-tqnwum-20250528125632 Cluster: redis-tqnwum Namespace: ns-rdwsc Spec: Method: volume-snapshot Policy Name: redis-tqnwum-redis-backup-policy Actions: createVolumeSnapshot-0: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2b1becf] goroutine 1 [running]: github.com/apecloud/kbcli/pkg/cmd/dataprotection.PrintBackupObjDescribe(0xc000672600, 0xc00067e588) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:480 +0x4cf github.com/apecloud/kbcli/pkg/cmd/dataprotection.DescribeBackups(0xc000672600, ***0xc0010a3950?, 0x18fd69b?, 0xc001368d88?***) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:458 +0x125 github.com/apecloud/kbcli/pkg/cmd/cluster.describeBackups(0x0?, ***0xc000750ec0?, 0x0?, 0x74b1bc5200000000?***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:204 +0x66 github.com/apecloud/kbcli/pkg/cmd/cluster.NewDescribeBackupCmd.func1(0xc001156608?, ***0xc000750ec0, 0x0, 0x4***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:195 +0xe5 github.com/spf13/cobra.(*Command).execute(0xc001156608, ***0xc000750e80, 0x4, 0x4***) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xab1 github.com/spf13/cobra.(*Command).ExecuteC(0xc000ef2308) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 k8s.io/component-base/cli.run(0xc000ef2308) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:146 +0x290 k8s.io/component-base/cli.RunNoErrOutput(...) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:84 main.main() /home/runner/work/kbcli/kbcli/cmd/cli/main.go:31 +0x18 cluster connect `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum-backup` set secret: redis-tqnwum-backup-redis-account-default `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-backup-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. # Server redis_version:7.2.4 redis_git_sha1:00000000 redis_git_dirty:0 redis_build_id:3a23a5ac67fe7008 redis_mode:standalone os:Linux 5.10.236-228.935.amzn2.x86_64 x86_64 arch_bits:64 monotonic_clock:POSIX clock_gettime multiplexing_api:epoll atomicvar_api:c11-builtin gcc_version:11.4.0 process_id:1 process_supervised:no run_id:fb7bf37083855abff6f1f61b6cdcb817cd208f78 tcp_port:6379 server_time_usec:1748408336674774 uptime_in_seconds:52 uptime_in_days:0 hz:10 configured_hz:10 lru_clock:3577872 executable:/redis-server config_file:/etc/redis/redis.conf io_threads_active:0 listener0:name=tcp,bind=*,bind=-::*,port=6379 `echo " echo \"CONFIG GET maxclients\" | redis-cli -h redis-tqnwum-backup-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-backup-redis-0 --namespace ns-rdwsc -- bash ` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. maxclients 10000 connect cluster Success set max_connections to 10000 delete cluster redis-tqnwum-backup `kbcli cluster delete redis-tqnwum-backup --auto-approve --namespace ns-rdwsc ` Cluster redis-tqnwum-backup deleted pod_info:redis-tqnwum-backup-redis-0 3/3 Running 0 79s redis-tqnwum-backup-redis-1 3/3 Running 0 52s redis-tqnwum-backup-redis-sentinel-0 2/2 Running 0 105s redis-tqnwum-backup-redis-sentinel-1 2/2 Running 0 106s redis-tqnwum-backup-redis-sentinel-2 2/2 Terminating 0 105s No resources found in ns-rdwsc namespace. delete cluster pod done No resources found in ns-rdwsc namespace. check cluster resource non-exist OK: pvc No resources found in ns-rdwsc namespace. delete cluster done No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-rdwsc-redis-tqnwum-20250528125632 --namespace ns-rdwsc ` backup.dataprotection.kubeblocks.io/backup-ns-rdwsc-redis-tqnwum-20250528125632 patched `kbcli cluster delete-backup redis-tqnwum --name backup-ns-rdwsc-redis-tqnwum-20250528125632 --force --auto-approve --namespace ns-rdwsc ` Backup backup-ns-rdwsc-redis-tqnwum-20250528125632 deleted No opsrequests found in ns-rdwsc namespace. cluster list-logs `kbcli cluster list-logs redis-tqnwum --namespace ns-rdwsc ` No log files found. Error from server (NotFound): pods "redis-tqnwum-redis-1" not found cluster logs `kbcli cluster logs redis-tqnwum --tail 30 --namespace ns-rdwsc ` Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) + sed '/user kbreplicator on/d' /data/users.acl + mv /data/users.acl.bak /data/users.acl + sed '/user kbreplicator-sentinel on/d' /data/users.acl + mv /data/users.acl.bak /data/users.acl + build_redis_default_accounts + unset_xtrace_when_ut_mode_false + '[' false == false ']' + set +x + echo 'aclfile /data/users.acl' build default accounts succeeded! Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so + echo 'build default accounts succeeded!' + start_redis_server + exec_cmd='exec redis-server /etc/redis/redis.conf' + '[' -f /opt/redis-stack/lib/redisearch.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so ' + '[' -f /opt/redis-stack/lib/redistimeseries.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so ' + '[' -f /opt/redis-stack/lib/rejson.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so ' + '[' -f /opt/redis-stack/lib/redisbloom.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so ' + '[' -f /opt/redis-stack/lib/redisgraph.so ']' + '[' -f /opt/redis-stack/lib/rediscompat.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so' + '[' -f /opt/redis-stack/lib/redisgears.so ']' + exec_cmd='exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so ' + echo 'Starting redis server cmd: exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so ' + eval 'exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so ' ++ exec redis-server /etc/redis/redis.conf --loadmodule /opt/redis-stack/lib/redisearch.so --loadmodule /opt/redis-stack/lib/redistimeseries.so --loadmodule /opt/redis-stack/lib/rejson.so --loadmodule /opt/redis-stack/lib/redisbloom.so --loadmodule /opt/redis-stack/lib/rediscompat.so --loadmodule /opt/redis-stack/lib/redisgears.so v8-plugin-path /opt/redis-stack/lib/libredisgears_v8_plugin.so cluster logs running `kbcli cluster logs redis-tqnwum --tail 30 --file-type=running --namespace ns-rdwsc ` 1:M 28 May 2025 04:37:35.077 * Diskless rdb transfer, done reading from pipe, 1 replicas still up. 1:M 28 May 2025 04:37:35.085 * Background RDB transfer terminated with success 1:M 28 May 2025 04:37:35.085 * Streamed RDB transfer with replica redis-tqnwum-redis-0.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming 1:M 28 May 2025 04:37:35.085 * Synchronization with replica redis-tqnwum-redis-0.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 succeeded 1:M 28 May 2025 04:39:16.404 * 100 changes in 300 seconds. Saving... 1:M 28 May 2025 04:39:16.405 * Background saving started by pid 745 745:C 28 May 2025 04:39:16.415 * DB saved on disk 745:C 28 May 2025 04:39:16.459 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB 1:M 28 May 2025 04:39:16.559 * Background saving terminated with success 1:M 28 May 2025 04:40:17.048 * 10000 changes in 60 seconds. Saving... 1:M 28 May 2025 04:40:17.049 * Background saving started by pid 843 843:C 28 May 2025 04:40:17.056 * DB saved on disk 843:C 28 May 2025 04:40:17.058 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB 1:M 28 May 2025 04:40:17.151 * Background saving terminated with success 1:M 28 May 2025 04:41:18.076 * 10000 changes in 60 seconds. Saving... 1:M 28 May 2025 04:41:18.078 * Background saving started by pid 949 949:C 28 May 2025 04:41:18.162 * DB saved on disk 949:C 28 May 2025 04:41:18.165 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB 1:M 28 May 2025 04:41:18.180 * Background saving terminated with success 1:M 28 May 2025 04:55:28.405 * Replica redis-tqnwum-redis-2.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 asks for synchronization 1:M 28 May 2025 04:55:28.405 * Full resync requested by replica redis-tqnwum-redis-2.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 1:M 28 May 2025 04:55:28.405 * Delay next BGSAVE for diskless SYNC 1:M 28 May 2025 04:55:33.013 * Starting BGSAVE for SYNC with target: replicas sockets 1:M 28 May 2025 04:55:33.015 * Background RDB transfer started by pid 2415 2415:C 28 May 2025 04:55:33.022 * Fork CoW for RDB: current 1 MB, peak 1 MB, average 1 MB 1:M 28 May 2025 04:55:33.022 * Diskless rdb transfer, done reading from pipe, 1 replicas still up. 1:M 28 May 2025 04:55:33.116 * Background RDB transfer terminated with success 1:M 28 May 2025 04:55:33.116 * Streamed RDB transfer with replica redis-tqnwum-redis-2.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 succeeded (socket). Waiting for REPLCONF ACK from replica to enable streaming 1:M 28 May 2025 04:55:33.116 * Synchronization with replica redis-tqnwum-redis-2.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 succeeded 1:M 28 May 2025 04:55:40.748 * Connection with replica redis-tqnwum-redis-0.redis-tqnwum-redis-headless.ns-rdwsc.svc.cluster.local:6379 lost. LB_TYPE is set to: internet cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose redis-tqnwum --auto-approve --force=true --type internet --enable false --components redis --role-selector primary --namespace ns-rdwsc ` OpsRequest redis-tqnwum-expose-297v7 created successfully, you can view the progress: kbcli cluster describe-ops redis-tqnwum-expose-297v7 -n ns-rdwsc check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-expose-297v7 ns-rdwsc Expose redis-tqnwum redis Running 0/1 May 28,2025 12:59 UTC+0800 check cluster status `kbcli cluster list redis-tqnwum --show-labels --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS redis-tqnwum ns-rdwsc redis WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=redis-tqnwum,clusterdefinition.kubeblocks.io/name=redis check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances redis-tqnwum --namespace ns-rdwsc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME redis-tqnwum-redis-1 ns-rdwsc redis-tqnwum redis Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:32 UTC+0800 redis-tqnwum-redis-2 ns-rdwsc redis-tqnwum redis Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:6Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:55 UTC+0800 redis-tqnwum-redis-sentinel-0 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-1 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-2-51.us-west-2.compute.internal/172.31.2.51 May 28,2025 12:23 UTC+0800 redis-tqnwum-redis-sentinel-2 ns-rdwsc redis-tqnwum redis-sentinel Running us-west-2a 100m / 100m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:23 UTC+0800 check pod status done check cluster role check cluster role done primary: redis-tqnwum-redis-1;secondary: redis-tqnwum-redis-2 `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check cluster connect `echo 'echo "" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr' | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash` check cluster connect done check ops status `kbcli cluster list-ops redis-tqnwum --status all --namespace ns-rdwsc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME redis-tqnwum-expose-297v7 ns-rdwsc Expose redis-tqnwum redis Succeed 1/1 May 28,2025 12:59 UTC+0800 check ops status done ops_status:redis-tqnwum-expose-297v7 ns-rdwsc Expose redis-tqnwum redis Succeed 1/1 May 28,2025 12:59 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations redis-tqnwum-expose-297v7 --namespace ns-rdwsc ` opsrequest.operations.kubeblocks.io/redis-tqnwum-expose-297v7 patched `kbcli cluster delete-ops --name redis-tqnwum-expose-297v7 --force --auto-approve --namespace ns-rdwsc ` OpsRequest redis-tqnwum-expose-297v7 deleted `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 `echo "echo \"get mykey\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-2 --namespace ns-rdwsc -- bash ` check readonly data: Defaulted container "redis" out of: redis, metrics, kbagent, init-dbctl (init), init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe. satbh check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h redis-tqnwum-redis-redis.ns-rdwsc.svc.cluster.local -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-1 --namespace ns-rdwsc -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=redis-tqnwum` set secret: redis-tqnwum-redis-account-default `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.username***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.password***"` `kubectl get secrets redis-tqnwum-redis-account-default -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:26379 DB_USERNAME:default;DB_PASSWORD:3L7022eJcr;DB_PORT:6379;DB_DATABASE:db0 check readonly db_client batch data count `echo "echo \"get executions_loop_key\" | redis-cli -h 127.0.0.1 -p 6379 -a 3L7022eJcr " | kubectl exec -it redis-tqnwum-redis-2 --namespace ns-rdwsc -- bash ` check readonly db_client batch data Success delete cluster redis-tqnwum `kbcli cluster delete redis-tqnwum --auto-approve --namespace ns-rdwsc ` Cluster redis-tqnwum deleted pod_info:redis-tqnwum-redis-1 3/3 Running 0 27m redis-tqnwum-redis-2 3/3 Running 0 5m7s redis-tqnwum-redis-sentinel-0 2/2 Running 0 37m redis-tqnwum-redis-sentinel-1 2/2 Running 0 37m redis-tqnwum-redis-sentinel-2 2/2 Running 0 37m No resources found in ns-rdwsc namespace. delete cluster pod done No resources found in ns-rdwsc namespace. check cluster resource non-exist OK: pvc No resources found in ns-rdwsc namespace. delete cluster done No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. No resources found in ns-rdwsc namespace. Redis Test Suite All Done! --------------------------------------Redis (Topology = replication Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=redis-7-1.0.0-alpha.0;ComponentVersion=redis;ServiceVersion=7.2.4;]|[Description=Create a cluster with the specified component definition redis-7-1.0.0-alpha.0 and component version redis and service version 7.2.4] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=satbh]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=satbh;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Expose]|[Enable=true;TYPE=internet;ComponentName=redis]|[Description=Expose Enable the internet service with redis component] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=redis]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[HorizontalScaling Out]|[ComponentName=redis]|[Description=HorizontalScaling Out the cluster specify component redis] [PASSED]|[HorizontalScaling In]|[ComponentName=redis]|[Description=HorizontalScaling In the cluster specify component redis] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=redis]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[VerticalScaling]|[ComponentName=redis]|[Description=VerticalScaling the cluster specify component redis] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=redis]|[Description=] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=redis]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=redis]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=redis]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Full CPU Failover;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=redis]|[Description=] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=redis]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [SKIPPED]|[Failover]|[HA=Evicting Pod;ComponentName=redis]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Reconfiguring]|[ComponentName=redis;maxclients=10001]|[Description=Reconfiguring the cluster specify component redis set maxclients=10001] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=redis]|[Description=] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=redis]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Bench]|[ComponentName=redis]|[Description=Bench the cluster service with redis component] [PASSED]|[Bench]|[HostType=LB;ComponentName=redis]|[Description=Bench the cluster LB service with redis component] [PASSED]|[VolumeExpansion]|[ComponentName=redis]|[Description=VolumeExpansion the cluster specify component redis] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=redis]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[Schedule=true;BackupMethod=datafile]|[Description=The cluster Schedule datafile Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=datafile]|[Description=The cluster Schedule datafile Restore] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=datafile]|[Description=Delete the Schedule datafile restore cluster] [PASSED]|[Backup]|[BackupMethod=datafile]|[Description=The cluster datafile Backup] [PASSED]|[Restore]|[BackupMethod=datafile]|[Description=The cluster datafile Restore] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=datafile]|[Description=Delete the datafile restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=redis]|[Description=Rebuild the cluster instance specify component redis] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=redis]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Expose]|[Disable=true;TYPE=internet;ComponentName=redis]|[Description=Expose Disable the internet service with redis component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]