bash test/kbcli/test_kbcli_0.9.sh --type 6 --version 0.9.5 --chaos-mesh true --drain-node true --aws-access-key-id *** --aws-secret-access-key *** --jihulab-token *** --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-thtcx ` `kubectl create namespace ns-thtcx` namespace/ns-thtcx created create namespace ns-thtcx done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 12 32.1M 12 4125k 0 0 8249k 0 0:00:03 --:--:-- 0:00:03 8249k 100 32.1M 100 32.1M 0 0 23.3M 0 0:00:01 0:00:01 --:--:-- 32.0M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:mongodb set component version set component version:mongodb set service versions:4.0.28,4.2.25,4.4.29,5.0.29,6.0.21,7.0.18,8.0.8 set service versions sorted:4.0.28,4.2.25,4.4.29,5.0.29,6.0.21,7.0.18,8.0.8 no cluster version found set mongodb component definition set mongodb component definition mongo-mongos REPORT_COUNT 0:0 set replicas first:3,4.0.28|3,4.2.25|3,4.4.29|3,5.0.29|3,6.0.21|3,7.0.18|3,8.0.8 set replicas third:3,4.0.28 set replicas fourth:3,4.0.28 set minimum cmpv service version set minimum cmpv service version replicas:3,4.0.28 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Not found topology in cluster definition mongodb LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 3 No resources found in ns-thtcx namespace. termination_policy:Delete create 3 replica Delete mongodb cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition2: mongodb by component version:mongodb apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: mongodb-sgqlyj namespace: ns-thtcx spec: terminationPolicy: Delete componentSpecs: - name: mongodb componentDef: mongodb serviceVersion: 4.0.28 replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi updateStrategy: BestEffortParallel volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi `kubectl apply -f test_create_mongodb-sgqlyj.yaml` cluster.apps.kubeblocks.io/mongodb-sgqlyj created apply test_create_mongodb-sgqlyj.yaml Success `rm -rf test_create_mongodb-sgqlyj.yaml` check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Aug 25,2025 20:46 UTC+0800 cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 20:46 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:50 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: check pod mongodb-sgqlyj-mongodb-0 container_name mongodb exist password h7B7m8Jz9786q3Ix Container mongodb logs contain secret password:2025-08-25T12:50:17Z INFO MongoDB Create user: root, passwd: h7B7m8Jz9786q3Ix, roles: map[db:admin role:root] describe cluster `kbcli cluster describe mongodb-sgqlyj --namespace ns-thtcx ` Name: mongodb-sgqlyj Created Time: Aug 25,2025 20:46 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-thtcx Running Delete Endpoints: COMPONENT MODE INTERNAL EXTERNAL mongodb ReadWrite mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local:27017 mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017 mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mongodb mongodb-sgqlyj-mongodb-0 primary Running 0 aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 20:46 UTC+0800 mongodb mongodb-sgqlyj-mongodb-1 secondary Running 0 aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:50 UTC+0800 mongodb mongodb-sgqlyj-mongodb-2 secondary Running 0 aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:51 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mongodb false 100m / 100m 512Mi / 512Mi data:3Gi default Images: COMPONENT TYPE IMAGE mongodb docker.io/apecloud/percona-server-mongodb:4.0.28 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-thtcx mongodb-sgqlyj `kbcli cluster label mongodb-sgqlyj app.kubernetes.io/instance- --namespace ns-thtcx ` label "app.kubernetes.io/instance" not found. `kbcli cluster label mongodb-sgqlyj app.kubernetes.io/instance=mongodb-sgqlyj --namespace ns-thtcx ` `kbcli cluster label mongodb-sgqlyj --list --namespace ns-thtcx ` NAME NAMESPACE LABELS mongodb-sgqlyj ns-thtcx app.kubernetes.io/instance=mongodb-sgqlyj label cluster app.kubernetes.io/instance=mongodb-sgqlyj Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=mongodb-sgqlyj --namespace ns-thtcx ` `kbcli cluster label mongodb-sgqlyj --list --namespace ns-thtcx ` NAME NAMESPACE LABELS mongodb-sgqlyj ns-thtcx app.kubernetes.io/instance=mongodb-sgqlyj case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label mongodb-sgqlyj case.name=kbcli.test2 --overwrite --namespace ns-thtcx ` `kbcli cluster label mongodb-sgqlyj --list --namespace ns-thtcx ` NAME NAMESPACE LABELS mongodb-sgqlyj ns-thtcx app.kubernetes.io/instance=mongodb-sgqlyj case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label mongodb-sgqlyj case.name- --namespace ns-thtcx ` `kbcli cluster label mongodb-sgqlyj --list --namespace ns-thtcx ` NAME NAMESPACE LABELS mongodb-sgqlyj ns-thtcx app.kubernetes.io/instance=mongodb-sgqlyj delete cluster label case.name Success list-accounts on characterType mongodb is not supported yet cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo " echo \"rs.status()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("c5925b45-21d1-4cf1-9550-379c0773c8e4") *** Percona Server for MongoDB server version: v4.0.28-23 *** "set" : "mongodb-sgqlyj-mongodb", "date" : ISODate("2025-08-25T12:52:42.224Z"), "myState" : 1, "term" : NumberLong(1), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : *** "lastCommittedOpTime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "readConcernMajorityOpTime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "appliedOpTime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "durableOpTime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) *** ***, "lastStableCheckpointTimestamp" : Timestamp(1756126330, 3), "electionCandidateMetrics" : *** "lastElectionReason" : "electionTimeout", "lastElectionDate" : ISODate("2025-08-25T12:50:11.959Z"), "electionTerm" : NumberLong(1), "lastCommittedOpTimeAtElection" : *** "ts" : Timestamp(0, 0), "t" : NumberLong(-1) ***, "lastSeenOpTimeAtElection" : *** "ts" : Timestamp(1756126211, 1), "t" : NumberLong(-1) ***, "numVotesNeeded" : 1, "priorityAtElection" : 2, "electionTimeoutMillis" : NumberLong(10000), "newTermStartDate" : ISODate("2025-08-25T12:50:11.962Z"), "wMajorityWriteAvailabilityDate" : ISODate("2025-08-25T12:50:12.076Z") ***, "members" : [ *** "_id" : 0, "name" : "mongodb-sgqlyj-mongodb-0.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 157, "optime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "optimeDate" : ISODate("2025-08-25T12:52:40Z"), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "", "electionTime" : Timestamp(1756126211, 2), "electionDate" : ISODate("2025-08-25T12:50:11Z"), "configVersion" : 3, "self" : true, "lastHeartbeatMessage" : "" ***, *** "_id" : 1, "name" : "mongodb-sgqlyj-mongodb-1.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 83, "optime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "optimeDurable" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "optimeDate" : ISODate("2025-08-25T12:52:40Z"), "optimeDurableDate" : ISODate("2025-08-25T12:52:40Z"), "lastHeartbeat" : ISODate("2025-08-25T12:52:40.923Z"), "lastHeartbeatRecv" : ISODate("2025-08-25T12:52:40.923Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "mongodb-sgqlyj-mongodb-0.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "syncSourceHost" : "mongodb-sgqlyj-mongodb-0.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 3 ***, *** "_id" : 2, "name" : "mongodb-sgqlyj-mongodb-2.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 43, "optime" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "optimeDurable" : *** "ts" : Timestamp(1756126360, 2), "t" : NumberLong(1) ***, "optimeDate" : ISODate("2025-08-25T12:52:40Z"), "optimeDurableDate" : ISODate("2025-08-25T12:52:40Z"), "lastHeartbeat" : ISODate("2025-08-25T12:52:41.123Z"), "lastHeartbeatRecv" : ISODate("2025-08-25T12:52:41.923Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "mongodb-sgqlyj-mongodb-0.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "syncSourceHost" : "mongodb-sgqlyj-mongodb-0.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 3 *** ], "ok" : 1, "operationTime" : Timestamp(1756126360, 2), "$clusterTime" : *** "clusterTime" : Timestamp(1756126360, 2), "signature" : *** "hash" : BinData(0,"C+zvuIujEfAzvdoWVxOxCYtYp8s="), "keyId" : NumberLong("7542504643893395459") *** *** *** bye connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-mongodb-sgqlyj" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mongodb-sgqlyj --namespace ns-thtcx ` Error from server (NotFound): pods "test-db-client-executionloop-mongodb-sgqlyj" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-mongodb-sgqlyj" not found `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-mongodb-sgqlyj namespace: ns-thtcx spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local" - "--user" - "root" - "--password" - "h7B7m8Jz9786q3Ix" - "--port" - "27017" - "--dbtype" - "mongodb" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-mongodb-sgqlyj.yaml` pod/test-db-client-executionloop-mongodb-sgqlyj created apply test-db-client-executionloop-mongodb-sgqlyj.yaml Success `rm -rf test-db-client-executionloop-mongodb-sgqlyj.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 21s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 26s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 31s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 37s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 42s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 48s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 58s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 1/1 Running 0 64s check pod test-db-client-executionloop-mongodb-sgqlyj status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-sgqlyj 0/1 Completed 0 69s check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 20:46 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:50 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1326d*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1326e*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1326f*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13270*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13271*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13272*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13273*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13274*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13275*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13276*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13277*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13278*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13279*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1327a*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1327b*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1327c*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1327d*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1327e*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1327f*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13280*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13281*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13282*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13283*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13284*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13285*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13286*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13287*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13288*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13289*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1328a*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1328b*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1328c*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1328d*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1328e*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b1328f*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13290*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13291*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13292*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13293*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13294*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13295*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13296*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13297*** Inserted document: BsonObjectId***value=68ac5cdbc490f57db6b13298*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b13299*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b1329a*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b1329b*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b1329c*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b1329d*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b1329e*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b1329f*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a0*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a1*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a2*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a3*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a4*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a5*** [ 59s ] executions total: 4242 successful: 4235 failed: 7 disconnect: 1 Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a6*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a7*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a8*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132a9*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132aa*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132ab*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132ac*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132ad*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132ae*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132af*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b0*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b1*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b2*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b3*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b4*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b5*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b6*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b7*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b8*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132b9*** Inserted document: BsonObjectId***value=68ac5cdcc490f57db6b132ba*** [ 60s ] executions total: 4263 successful: 4256 failed: 7 disconnect: 1 12:53:48.655 [main] INFO org.mongodb.driver.connection - Opened connection [connectionId***localValue:25, serverValue:1327***] to mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local:27017 Test Result: Total Executions: 4263 Successful Executions: 4256 Failed Executions: 7 Disconnection Counts: 1 Connection Information: Database Type: mongodb Host: mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local Port: 27017 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 4256 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mongodb-sgqlyj --namespace ns-thtcx ` pod/test-db-client-executionloop-mongodb-sgqlyj patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-mongodb-sgqlyj" force deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.remove(***) ; db.col.insertOne(***a:'feqkr'***)\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("05958583-3e8e-4b3f-90a1-cdd4e3bd52de") *** Percona Server for MongoDB server version: v4.0.28-23 *** "acknowledged" : true, "insertedId" : ObjectId("68ac5cf7945ab794d9bf34a7") *** bye add consistent data feqkr Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.remove(***) ; db.col.insertOne(***a:'feqkr'***)\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("dedc0b16-bfe1-4984-a3d3-bdcb311ae6f0") *** Percona Server for MongoDB server version: v4.0.28-23 2025-08-25T12:54:33.269+0000 E QUERY [js] WriteCommandError: not master : WriteCommandError(*** "operationTime" : Timestamp(1756126470, 2), "ok" : 0, "errmsg" : "not master", "code" : 10107, "codeName" : "NotMaster", "$clusterTime" : *** "clusterTime" : Timestamp(1756126470, 2), "signature" : *** "hash" : BinData(0,"DIiXDBs4WiTTglYMQXXSSBOBEYk="), "keyId" : NumberLong("7542504643893395459") *** *** ***) WriteCommandError@src/mongo/shell/bulk_api.js:420:48 Bulk/executeBatch@src/mongo/shell/bulk_api.js:902:1 Bulk/this.execute@src/mongo/shell/bulk_api.js:1150:21 DBCollection.prototype.insertOne@src/mongo/shell/crud_api.js:252:9 @(shell):1:21 bye command terminated with exit code 1 check add consistent data readonly Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running check node drain check node drain success `kill 1` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 20:46 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:50 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("9ef06e03-3332-417b-b11e-263627439197") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("bb79279e-002d-469f-933d-cedcd8b9b7b9") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 action: partition mode: all target: mode: all selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-mongodb-sgqlyj created apply test-chaos-mesh-networkpartition-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networkpartition-mongodb-sgqlyj.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 20:46 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:50 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("f91b0caf-79bd-492f-815d-c66bb30ec14c") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("ece5eed5-e1cc-4ec7-9f45-6132e212135b") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-mongodb-sgqlyj created apply test-chaos-mesh-networkduplicate-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networkduplicate-mongodb-sgqlyj.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 20:46 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:50 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 20:51 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("2b7d7f6a-d92c-4638-9025-0cd0ece138a7") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("da7f9b3d-7158-4f93-9413-c3d4b2eefdec") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart mongodb-sgqlyj --auto-approve --force=true --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-restart-67vg9 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-restart-67vg9 -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-restart-67vg9 ns-thtcx Restart mongodb-sgqlyj mongodb Running 0/3 Aug 25,2025 21:00 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-restart-67vg9 ns-thtcx Restart mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:00 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-restart-67vg9 ns-thtcx Restart mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:00 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-restart-67vg9 --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-restart-67vg9 patched `kbcli cluster delete-ops --name mongodb-sgqlyj-restart-67vg9 --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-restart-67vg9 deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("deabd277-86e7-4b85-aa43-b4b3598f073c") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("2a153ac8-b934-4ac9-88c3-08358ca71824") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-sgqlyj" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-2 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-mongodb-sgqlyj.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-mongodb-sgqlyj created apply test-chaos-mesh-podfailure-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-podfailure-mongodb-sgqlyj.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Abnormal Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-sgqlyj" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("0aae07fa-5372-4166-ac9b-966a1799f557") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("417a6f4f-2c02-41d3-ab0b-2ead46e43fde") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster update enable-all-logs `kbcli cluster update mongodb-sgqlyj --enable-all-logs=true --namespace ns-thtcx ` error: name is required check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done skip cluster Upgrade test failover fullcpu check cluster status before cluster-failover-fullcpu check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-sgqlyj" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-mongodb-sgqlyj.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-mongodb-sgqlyj created apply test-chaos-mesh-fullcpu-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-fullcpu-mongodb-sgqlyj.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-sgqlyj" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-0 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("6626bbdd-3acd-46a9-ac83-4921a1f5c175") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("a945fd86-acea-4bcb-a9d6-aaf20c381c9a") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-mongodb-sgqlyj" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-mongodb-sgqlyj.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-mongodb-sgqlyj created apply test-chaos-mesh-podkill-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-podkill-mongodb-sgqlyj.yaml` check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-mongodb-sgqlyj" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-mongodb-sgqlyj patched check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover podkill Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("6aa87196-c9aa-4df8-949a-a87270306344") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("77c22a4c-bdf1-43f8-9680-3669ae0558c5") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster rebulid instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-sgqlyj-rebuildinstance- namespace: ns-thtcx spec: type: RebuildInstance clusterName: mongodb-sgqlyj force: true rebuildFrom: - componentName: mongodb instances: - name: mongodb-sgqlyj-mongodb-0 inPlace: false check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-sgqlyj.yaml` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-rebuildinstance-xvp2f created create test_ops_cluster_mongodb-sgqlyj.yaml Success `rm -rf test_ops_cluster_mongodb-sgqlyj.yaml` check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` No opsrequests found in ns-thtcx namespace. No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: No opsrequests found in ns-thtcx namespace. ops_status: not found ops rebuildinstance check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("0a31e826-8e53-4bf2-a2ed-53949012806c") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("0d66ae64-744d-4e87-8e18-72c764e90197") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-sgqlyj namespace. `kbcli cluster hscale mongodb-sgqlyj --auto-approve --force=true --components mongodb --replicas 4 --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-horizontalscaling-2t72f created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-horizontalscaling-2t72f -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-horizontalscaling-2t72f ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:14 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 mongodb-sgqlyj-mongodb-3 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-default-15381528-vmss000000/10.224.0.4 Aug 25,2025 21:14 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 mongodb-sgqlyj-mongodb-3 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done No resources found in mongodb-sgqlyj namespace. check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-horizontalscaling-2t72f ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:14 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-horizontalscaling-2t72f ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:14 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-horizontalscaling-2t72f --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-horizontalscaling-2t72f patched `kbcli cluster delete-ops --name mongodb-sgqlyj-horizontalscaling-2t72f --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-horizontalscaling-2t72f deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("34cb922f-5c4d-4864-b044-4e5f9e7b3443") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("7b7da69e-79a4-4bea-9bc8-6bd6daf970ad") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-sgqlyj namespace. `kbcli cluster hscale mongodb-sgqlyj --auto-approve --force=true --components mongodb --replicas 3 --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-horizontalscaling-v4mj5 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-horizontalscaling-v4mj5 -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-horizontalscaling-v4mj5 ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:16 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done No resources found in mongodb-sgqlyj namespace. check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-horizontalscaling-v4mj5 ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:16 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-horizontalscaling-v4mj5 ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-horizontalscaling-v4mj5 --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-horizontalscaling-v4mj5 patched `kbcli cluster delete-ops --name mongodb-sgqlyj-horizontalscaling-v4mj5 --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-horizontalscaling-v4mj5 deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("05172022-b3dc-4424-9512-e794ce427fee") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("79352f27-0194-414b-a61b-016d9f13326c") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-sgqlyj" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-mongodb-sgqlyj.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-mongodb-sgqlyj created apply test-chaos-mesh-timeoffset-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-timeoffset-mongodb-sgqlyj.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:01 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` connect checking... check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-sgqlyj" force deleted timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-mongodb-sgqlyj patched check failover pod name failover pod name:mongodb-sgqlyj-mongodb-2 failover timeoffset Failure `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("df3f89ca-0f6a-4863-a01e-45d34596b293") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("4bf20f3e-dba6-485a-a1e3-77e0a800767f") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=mongodb-sgqlyj,apps.kubeblocks.io/component-name=mongodb,apps.kubeblocks.io/vct-name=data --namespace ns-thtcx ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-sgqlyj namespace. `kbcli cluster volume-expand mongodb-sgqlyj --auto-approve --force=true --components mongodb --volume-claim-templates data --storage 6Gi --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-volumeexpansion-82zsk created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-volumeexpansion-82zsk -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-volumeexpansion-82zsk ns-thtcx VolumeExpansion mongodb-sgqlyj mongodb Creating -/- Aug 25,2025 21:19 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:23 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:22 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done No resources found in mongodb-sgqlyj namespace. check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-volumeexpansion-82zsk ns-thtcx VolumeExpansion mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:19 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-volumeexpansion-82zsk ns-thtcx VolumeExpansion mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-volumeexpansion-82zsk --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-volumeexpansion-82zsk patched `kbcli cluster delete-ops --name mongodb-sgqlyj-volumeexpansion-82zsk --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-volumeexpansion-82zsk deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("df656426-456f-4533-b6f9-c34ec4e691ad") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("98bc0a1c-ae38-42cd-aff8-0eb4f1b4ba72") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-sgqlyj" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-mongodb-sgqlyj.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-mongodb-sgqlyj created apply test-chaos-mesh-dnserror-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-dnserror-mongodb-sgqlyj.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:23 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:22 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-sgqlyj" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("5d9e5383-cb4b-43b8-b321-5805f9287f32") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("c2586505-3018-4f15-b16b-0a76705b5876") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `mongo ***mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg:Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/?gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("f58881f0-e6bc-44a0-88ba-915567f39eaf") *** Percona Server for MongoDB server version: v4.0.28-23 bye connect headlessEndpoints Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-sgqlyj" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mongodb-sgqlyj.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mongodb-sgqlyj created apply test-chaos-mesh-oom-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-oom-mongodb-sgqlyj.yaml` check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:09 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:23 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:22 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-sgqlyj" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mongodb-sgqlyj patched check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("bc4edfa5-8d0b-4efc-a462-23917b1011d0") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("16ad1643-89b7-4acb-8f6b-8ce4458ba7f2") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster configure component_tmp: mongodb check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure mongodb-sgqlyj --auto-approve --force=true --set systemLog.quiet=false,systemLog.verbosity=1 --components mongodb --config-spec mongodb-config --config-file mongodb.conf --namespace ns-thtcx ` Will updated configure file meta: ConfigSpec: mongodb-config ConfigFile: mongodb.conf ComponentName: mongodb ClusterName: mongodb-sgqlyj OpsRequest mongodb-sgqlyj-reconfiguring-nn865 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-reconfiguring-nn865 -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-reconfiguring-nn865 ns-thtcx Reconfiguring mongodb-sgqlyj mongodb Running -/- Aug 25,2025 21:31 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:34 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:33 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:31 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-reconfiguring-nn865 ns-thtcx Reconfiguring mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:31 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-reconfiguring-nn865 ns-thtcx Reconfiguring mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-reconfiguring-nn865 --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-reconfiguring-nn865 patched `kbcli cluster delete-ops --name mongodb-sgqlyj-reconfiguring-nn865 --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-reconfiguring-nn865 deleted check config variables `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: db.adminCommand('getCmdLineOpts') mongosh --host mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file bash: line 1: mongosh: command not found command terminated with exit code 127 SERVICE_VERSION:4.0.28 Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: false configure:[systemLog.quiet] result actual:[false] equal expected:[false] `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: db.adminCommand('getCmdLineOpts') mongosh --host mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file bash: line 1: mongosh: command not found command terminated with exit code 127 SERVICE_VERSION:4.0.28 Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 1 configure:[systemLog.verbosity] result actual:[1] equal expected:[1] `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("6b36e85e-769b-4d3f-bcb2-e3c21aed5d55") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("71c451c6-b762-4be2-98b2-c9b2a0fae6f3") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj created apply test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:34 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:33 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:31 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj created apply test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:34 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:33 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:31 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-sgqlyj" not found check failover pod name:mongodb-sgqlyj-mongodb-0 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("64fd70ab-fe40-4202-848a-baf329d815b9") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("feb76768-97a5-4632-9487-f311e29c2a44") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote mongodb-sgqlyj --auto-approve --force=true --component mongodb --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-custom-4th5b created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-custom-4th5b -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-custom-4th5b ns-thtcx Custom mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:42 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:34 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:33 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:31 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-custom-4th5b ns-thtcx Custom mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:42 UTC+0800 ops_status:mongodb-sgqlyj-custom-4th5b ns-thtcx Custom mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:42 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-custom-4th5b ns-thtcx Custom mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:42 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-custom-4th5b --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-custom-4th5b patched `kbcli cluster delete-ops --name mongodb-sgqlyj-custom-4th5b --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-custom-4th5b deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("891ab9ad-1dfb-4ccf-9d1e-3a496bc2ffeb") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("f10ea7bb-dbc0-4fc6-9049-479a855a99af") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success switchover pod:mongodb-sgqlyj-mongodb-2 switchover success cluster configure component_tmp: mongodb check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure mongodb-sgqlyj --auto-approve --force=true --set systemLog.quiet=true --components mongodb --config-spec mongodb-config --config-file mongodb.conf --namespace ns-thtcx ` Will updated configure file meta: ConfigSpec: mongodb-config ConfigFile: mongodb.conf ComponentName: mongodb ClusterName: mongodb-sgqlyj OpsRequest mongodb-sgqlyj-reconfiguring-ktdjg created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-reconfiguring-ktdjg -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-reconfiguring-ktdjg ns-thtcx Reconfiguring mongodb-sgqlyj mongodb Creating -/- Aug 25,2025 21:43 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:45 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:43 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:44 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-reconfiguring-ktdjg ns-thtcx Reconfiguring mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:43 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-reconfiguring-ktdjg ns-thtcx Reconfiguring mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-reconfiguring-ktdjg --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-reconfiguring-ktdjg patched `kbcli cluster delete-ops --name mongodb-sgqlyj-reconfiguring-ktdjg --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-reconfiguring-ktdjg deleted check config variables `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: db.adminCommand('getCmdLineOpts') mongosh --host mongodb-sgqlyj-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file bash: line 1: mongosh: command not found command terminated with exit code 127 SERVICE_VERSION:4.0.28 Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: true configure:[systemLog.quiet] result actual:[true] equal expected:[true] `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("810de769-3daa-4955-9f71-dba16359be88") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("2a028405-401d-4541-b38c-8de39f382029") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-sgqlyj" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-2 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-mongodb-sgqlyj.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-mongodb-sgqlyj created apply test-chaos-mesh-dnsrandom-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-dnsrandom-mongodb-sgqlyj.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:45 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:43 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:44 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-sgqlyj" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-2 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("8c2db000-6107-4f9e-b4b1-31aeb54a8cbc") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("0083fb23-c716-440b-8a2b-261728b87549") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success cmpv upgrade service version:3,4.0.28|3,4.2.25|3,4.4.29|3,5.0.29|3,6.0.21|3,7.0.18|3,8.0.8 set latest cmpv service version latest service version:4.0.28 cmpv service version upgrade and downgrade cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale mongodb-sgqlyj --auto-approve --force=true --components mongodb --cpu 200m --memory 0.6Gi --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-verticalscaling-hflzq created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-verticalscaling-hflzq -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-verticalscaling-hflzq ns-thtcx VerticalScaling mongodb-sgqlyj mongodb Running 0/3 Aug 25,2025 21:49 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:50 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:49 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:49 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-verticalscaling-hflzq ns-thtcx VerticalScaling mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:49 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-verticalscaling-hflzq ns-thtcx VerticalScaling mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:49 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-verticalscaling-hflzq --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-verticalscaling-hflzq patched `kbcli cluster delete-ops --name mongodb-sgqlyj-verticalscaling-hflzq --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-verticalscaling-hflzq deleted get mongodb primary pod name... `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("f62ff63c-2ef3-494e-a8ef-d5888a504c81") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("3be8bc1d-637c-4fec-b002-2dbe9bdccc31") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster hscale offline instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-sgqlyj-hscaleoffinstance- labels: app.kubernetes.io/instance: mongodb-sgqlyj app.kubernetes.io/managed-by: kubeblocks namespace: ns-thtcx spec: type: HorizontalScaling clusterName: mongodb-sgqlyj force: true horizontalScaling: - componentName: mongodb scaleIn: onlineInstancesToOffline: - mongodb-sgqlyj-mongodb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-sgqlyj.yaml` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-hscaleoffinstance-6mj8g created create test_ops_cluster_mongodb-sgqlyj.yaml Success `rm -rf test_ops_cluster_mongodb-sgqlyj.yaml` check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-hscaleoffinstance-6mj8g ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:51 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:49 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:49 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-hscaleoffinstance-6mj8g ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:51 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-hscaleoffinstance-6mj8g ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:51 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-hscaleoffinstance-6mj8g --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-hscaleoffinstance-6mj8g patched `kbcli cluster delete-ops --name mongodb-sgqlyj-hscaleoffinstance-6mj8g --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-hscaleoffinstance-6mj8g deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("0d123987-5809-4018-bff3-9351d41b618f") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("f4dafc45-167a-42d6-af8c-084b6d5c62f0") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster hscale online instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-sgqlyj-hscaleoninstance- labels: app.kubernetes.io/instance: mongodb-sgqlyj app.kubernetes.io/managed-by: kubeblocks namespace: ns-thtcx spec: type: HorizontalScaling clusterName: mongodb-sgqlyj force: true horizontalScaling: - componentName: mongodb scaleOut: offlineInstancesToOnline: - mongodb-sgqlyj-mongodb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-sgqlyj.yaml` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-hscaleoninstance-bt94k created create test_ops_cluster_mongodb-sgqlyj.yaml Success `rm -rf test_ops_cluster_mongodb-sgqlyj.yaml` check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-hscaleoninstance-bt94k ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Running 0/1 Aug 25,2025 21:51 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:51 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:49 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:49 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-hscaleoninstance-bt94k ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:51 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-hscaleoninstance-bt94k ns-thtcx HorizontalScaling mongodb-sgqlyj mongodb Succeed 1/1 Aug 25,2025 21:51 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-hscaleoninstance-bt94k --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-hscaleoninstance-bt94k patched `kbcli cluster delete-ops --name mongodb-sgqlyj-hscaleoninstance-bt94k --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-hscaleoninstance-bt94k deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("63012c74-16ca-4763-8d85-30fab03732f9") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("e2c8ee97-5817-4f2d-8125-5555a9de126d") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop mongodb-sgqlyj --auto-approve --force=true --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-stop-hdr8g created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-stop-hdr8g -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-stop-hdr8g ns-thtcx Stop mongodb-sgqlyj mongodb Running 0/3 Aug 25,2025 21:55 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Stopped Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-stop-hdr8g ns-thtcx Stop mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:55 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-stop-hdr8g ns-thtcx Stop mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:55 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-stop-hdr8g --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-stop-hdr8g patched `kbcli cluster delete-ops --name mongodb-sgqlyj-stop-hdr8g --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-stop-hdr8g deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start mongodb-sgqlyj --force=true --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-start-fxwhv created successfully, you can view the progress: kbcli cluster describe-ops mongodb-sgqlyj-start-fxwhv -n ns-thtcx check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-start-fxwhv ns-thtcx Start mongodb-sgqlyj mongodb Running 0/3 Aug 25,2025 21:55 UTC+0800 check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:56 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-sgqlyj --status all --namespace ns-thtcx ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-sgqlyj-start-fxwhv ns-thtcx Start mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:55 UTC+0800 check ops status done ops_status:mongodb-sgqlyj-start-fxwhv ns-thtcx Start mongodb-sgqlyj mongodb Succeed 3/3 Aug 25,2025 21:55 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-sgqlyj-start-fxwhv --namespace ns-thtcx ` opsrequest.apps.kubeblocks.io/mongodb-sgqlyj-start-fxwhv patched `kbcli cluster delete-ops --name mongodb-sgqlyj-start-fxwhv --force --auto-approve --namespace ns-thtcx ` OpsRequest mongodb-sgqlyj-start-fxwhv deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("eef4550c-699d-4638-a8de-b80ddbfe24ec") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("79ac493c-4421-4cbb-be64-345a03a651ec") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mongodb-sgqlyj created apply test-chaos-mesh-networkcorruptover-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-mongodb-sgqlyj.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:56 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-2 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("6f2e81c6-dcb6-439c-978d-679646aeed82") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("b0ffd919-1a2f-4b0a-853f-1d16e6a5c0de") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster update monitor cluster does not need to check monitor currently check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:56 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-2 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-mongodb-sgqlyj created apply test-chaos-mesh-networkdelay-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networkdelay-mongodb-sgqlyj.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000005/10.224.0.5 Aug 25,2025 21:56 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-2;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-2 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("f47efad1-5065-483a-b97b-bcafe8e69089") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("d2d452bf-86ed-46a3-aba3-bb8ca9f3dfe1") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover drainnode check cluster status before cluster-failover-drainnode check cluster status done cluster_status:Running check node drain check node drain success kubectl get pod mongodb-sgqlyj-mongodb-2 --namespace ns-thtcx -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-24318931-vmss000005 success check if multiple pods are on the same node kubectl get pod mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-24318931-vmss000006 success kubectl get pod mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-24318931-vmss000003 success kubectl drain aks-cicdamdpool-24318931-vmss000005 --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/aks-cicdamdpool-24318931-vmss000005 cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-7q7km, kb-ibmnm/kb-addon-apecloud-otel-collector-29kdm, kube-system/azure-cns-rftvf, kube-system/azure-ip-masq-agent-zz4xk, kube-system/cloud-node-manager-9c5kp, kube-system/csi-azuredisk-node-ckbmx, kube-system/csi-azurefile-node-sdcpj, kube-system/kube-proxy-9q8wr evicting pod ns-thtcx/mongodb-sgqlyj-mongodb-2 evicting pod chaos-mesh/chaos-dns-server-cd7c67cc5-5wzsz evicting pod chaos-mesh/chaos-controller-manager-74b8999f4-478lj evicting pod kb-ibmnm/kbcli-test-minio-6fc9947768-xjcgf evicting pod kb-ibmnm/kb-addon-grafana-84fcf4686f-kwbn9 evicting pod chaos-mesh/chaos-dashboard-7b866cc765-5kbwq evicting pod kb-ibmnm/kb-addon-prometheus-server-0 evicting pod ns-otdrv/redis-tdnvse-backup-redis-sentinel-0 evicting pod kb-ibmnm/keda-operator-68b7db459f-5sn2f evicting pod kube-system/konnectivity-agent-5cdb496446-7bpzt evicting pod kb-ibmnm/keda-operator-metrics-apiserver-68dcc49f6-qk4jj evicting pod ns-otdrv/redis-tdnvse-redis-twemproxy-2 evicting pod ns-shlpp/apemysql-noxclr-mysql-0 evicting pod kb-ibmnm/kubeblocks-6c89f94c9b-m4xr8 evicting pod ns-otdrv/redis-tdnvse-redis-sentinel-1 evicting pod chaos-mesh/chaos-controller-manager-74b8999f4-9pdlb evicting pod kb-ibmnm/kubeblocks-dataprotection-b7547b5bf-xx8s6 evicting pod kube-system/coredns-6f776c8fb5-pc84x pod/chaos-controller-manager-74b8999f4-478lj evicted pod/apemysql-noxclr-mysql-0 evicted pod/redis-tdnvse-redis-twemproxy-2 evicted pod/kubeblocks-6c89f94c9b-m4xr8 evicted pod/kb-addon-prometheus-server-0 evicted pod/redis-tdnvse-backup-redis-sentinel-0 evicted pod/chaos-dashboard-7b866cc765-5kbwq evicted pod/konnectivity-agent-5cdb496446-7bpzt evicted pod/kbcli-test-minio-6fc9947768-xjcgf evicted pod/mongodb-sgqlyj-mongodb-2 evicted pod/keda-operator-68b7db459f-5sn2f evicted pod/redis-tdnvse-redis-sentinel-1 evicted pod/chaos-controller-manager-74b8999f4-9pdlb evicted pod/kubeblocks-dataprotection-b7547b5bf-xx8s6 evicted pod/coredns-6f776c8fb5-pc84x evicted I0825 22:02:58.031622 24289 request.go:697] Waited for 1.20015187s due to client-side throttling, not priority and fairness, request: POST:https://more-lab-k8s-2dykbzzx.hcp.eastus.azmk8s.io:443/api/v1/namespaces/kb-ibmnm/pods/kb-addon-grafana-84fcf4686f-kwbn9/eviction pod/kb-addon-grafana-84fcf4686f-kwbn9 evicted pod/chaos-dns-server-cd7c67cc5-5wzsz evicted pod/keda-operator-metrics-apiserver-68dcc49f6-qk4jj evicted node/aks-cicdamdpool-24318931-vmss000005 drained kubectl uncordon aks-cicdamdpool-24318931-vmss000005 node/aks-cicdamdpool-24318931-vmss000005 uncordoned check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 22:03 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-1;secondary: mongodb-sgqlyj-mongodb-0 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash` check cluster connect done check failover pod name failover pod name:mongodb-sgqlyj-mongodb-1 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("79c06939-fc8e-41c6-b6dc-87de1cd71ebc") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("0dc9769d-4f2c-450a-8d2f-d3722c86b7a5") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check db_client batch data Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-sgqlyj" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-sgqlyj" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-mongodb-sgqlyj namespace: ns-thtcx spec: selector: namespaces: - ns-thtcx labelSelectors: apps.kubeblocks.io/pod-name: mongodb-sgqlyj-mongodb-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-mongodb-sgqlyj.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-mongodb-sgqlyj created apply test-chaos-mesh-networklossover-mongodb-sgqlyj.yaml Success `rm -rf test-chaos-mesh-networklossover-mongodb-sgqlyj.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx Delete Updating Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 22:03 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mongodb-sgqlyj --namespace ns-thtcx ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-sgqlyj" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-sgqlyj" not found check failover pod name failover pod name:mongodb-sgqlyj-mongodb-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find()\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("9b240a1c-cec4-444a-a688-1352c45635c4") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongo --host mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-1 --namespace ns-thtcx -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb-ro.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("b52e84ba-d2c6-4825-8c54-bc2923d4d3d4") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac5cf7945ab794d9bf34a7"), "a" : "feqkr" *** bye check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash ` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update mongodb-sgqlyj --termination-policy=WipeOut --namespace ns-thtcx ` cluster.apps.kubeblocks.io/mongodb-sgqlyj updated check cluster status `kbcli cluster list mongodb-sgqlyj --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj ns-thtcx WipeOut Running Aug 25,2025 20:46 UTC+0800 app.kubernetes.io/instance=mongodb-sgqlyj check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-mongodb-0 ns-thtcx mongodb-sgqlyj mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-1 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 21:55 UTC+0800 mongodb-sgqlyj-mongodb-2 ns-thtcx mongodb-sgqlyj mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 22:03 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-mongodb-0;secondary: mongodb-sgqlyj-mongodb-1 mongodb-sgqlyj-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done skip cluster datafile backup skip cluster volume-snapshot backup `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `db.msg.drop();db.createCollection('msg');db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj0',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj1',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj2',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj3',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj4',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj5',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj6',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj7',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj8',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj9',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj10',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj11',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj12',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj13',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj14',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-sgqlyj15',time: new Date()***);` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file bash: line 1: mongosh: command not found command terminated with exit code 127 Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("b2217685-68b8-4ea6-80ff-62ef9431fd8e") *** Percona Server for MongoDB server version: v4.0.28-23 *** "acknowledged" : true, "insertedId" : ObjectId("68ac6e18be1c41abe53a001f") *** bye cluster dump backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-z6rvb -n kb-ibmnm -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-z6rvb -n kb-ibmnm -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-ibmnm get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-ibmnm -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-ibmnm -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-ibmnm.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-ibmnm.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-6fc9947768-r9trk --namespace kb-ibmnm -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mongodb-sgqlyj --method dump --namespace ns-thtcx ` Backup backup-ns-thtcx-mongodb-sgqlyj-20250825220727 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-thtcx-mongodb-sgqlyj-20250825220727 -n ns-thtcx check backup status `kbcli cluster list-backups mongodb-sgqlyj --namespace ns-thtcx ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-thtcx-mongodb-sgqlyj-20250825220727 ns-thtcx mongodb-sgqlyj dump Aug 25,2025 22:07 UTC+0800 backup_status:mongodb-sgqlyj-dump-Running backup_status:mongodb-sgqlyj-dump-Running backup_status:mongodb-sgqlyj-dump-Running backup_status:mongodb-sgqlyj-dump-Running backup_status:mongodb-sgqlyj-dump-Running check backup status done backup_status:backup-ns-thtcx-mongodb-sgqlyj-20250825220727 ns-thtcx mongodb-sgqlyj dump Completed 50025 22s Aug 25,2025 22:07 UTC+0800 Aug 25,2025 22:07 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "mongodb-sgqlyj-backup" not found `kbcli cluster describe-backup backup-ns-thtcx-mongodb-sgqlyj-20250825220727 --namespace ns-thtcx ` Name: backup-ns-thtcx-mongodb-sgqlyj-20250825220727 Cluster: mongodb-sgqlyj Namespace: ns-thtcx Spec: Method: dump Policy Name: mongodb-sgqlyj-mongodb-backup-policy Status: Phase: Completed Total Size: 50025 ActionSet Name: mongodb-dump Repository: backuprepo-kbcli-test Duration: 22s Start Time: Aug 25,2025 22:07 UTC+0800 Completion Time: Aug 25,2025 22:07 UTC+0800 Path: /ns-thtcx/mongodb-sgqlyj-7aee63e2-f7ae-4f54-bd9d-d06275d54cb7/mongodb/backup-ns-thtcx-mongodb-sgqlyj-20250825220727 Time Range Start: Aug 25,2025 22:07 UTC+0800 Time Range End: Aug 25,2025 22:07 UTC+0800 Warning Events: `kbcli cluster restore mongodb-sgqlyj-backup --backup backup-ns-thtcx-mongodb-sgqlyj-20250825220727 --namespace ns-thtcx ` Cluster mongodb-sgqlyj-backup created check cluster status `kbcli cluster list mongodb-sgqlyj-backup --show-labels --namespace ns-thtcx ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-sgqlyj-backup ns-thtcx WipeOut Aug 25,2025 22:07 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-sgqlyj-backup --namespace ns-thtcx ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-sgqlyj-backup-mongodb-0 ns-thtcx mongodb-sgqlyj-backup mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 22:07 UTC+0800 mongodb-sgqlyj-backup-mongodb-1 ns-thtcx mongodb-sgqlyj-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000003/10.224.0.10 Aug 25,2025 22:08 UTC+0800 mongodb-sgqlyj-backup-mongodb-2 ns-thtcx mongodb-sgqlyj-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-24318931-vmss000006/10.224.0.7 Aug 25,2025 22:08 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-sgqlyj-backup-mongodb-0;secondary: mongodb-sgqlyj-backup-mongodb-1 mongodb-sgqlyj-backup-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj-backup` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. check cluster connect `echo "mongo --host mongodb-sgqlyj-backup-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-backup-mongodb-0 --namespace ns-thtcx -- bash` check cluster connect done `kbcli cluster describe-backup backup-ns-thtcx-mongodb-sgqlyj-20250825220727 --namespace ns-thtcx ` Name: backup-ns-thtcx-mongodb-sgqlyj-20250825220727 Cluster: mongodb-sgqlyj Namespace: ns-thtcx Spec: Method: dump Policy Name: mongodb-sgqlyj-mongodb-backup-policy Status: Phase: Completed Total Size: 50025 ActionSet Name: mongodb-dump Repository: backuprepo-kbcli-test Duration: 22s Start Time: Aug 25,2025 22:07 UTC+0800 Completion Time: Aug 25,2025 22:07 UTC+0800 Path: /ns-thtcx/mongodb-sgqlyj-7aee63e2-f7ae-4f54-bd9d-d06275d54cb7/mongodb/backup-ns-thtcx-mongodb-sgqlyj-20250825220727 Time Range Start: Aug 25,2025 22:07 UTC+0800 Time Range End: Aug 25,2025 22:07 UTC+0800 Warning Events: `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj-backup` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `db.msg.find();` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file bash: line 1: mongosh: command not found command terminated with exit code 127 Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-backup-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("30e2e512-b2b4-4cda-aa03-4331748d86c9") *** Percona Server for MongoDB server version: v4.0.28-23 *** "_id" : ObjectId("68ac6e18be1c41abe53a0017"), "msg" : "kbcli-test-data-sgqlyj7", "time" : ISODate("2025-08-25T14:07:20.024Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0010"), "msg" : "kbcli-test-data-sgqlyj0", "time" : ISODate("2025-08-25T14:07:20.020Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0013"), "msg" : "kbcli-test-data-sgqlyj3", "time" : ISODate("2025-08-25T14:07:20.022Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0015"), "msg" : "kbcli-test-data-sgqlyj5", "time" : ISODate("2025-08-25T14:07:20.023Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0018"), "msg" : "kbcli-test-data-sgqlyj8", "time" : ISODate("2025-08-25T14:07:20.024Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0012"), "msg" : "kbcli-test-data-sgqlyj2", "time" : ISODate("2025-08-25T14:07:20.022Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0011"), "msg" : "kbcli-test-data-sgqlyj1", "time" : ISODate("2025-08-25T14:07:20.021Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0014"), "msg" : "kbcli-test-data-sgqlyj4", "time" : ISODate("2025-08-25T14:07:20.022Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a001c"), "msg" : "kbcli-test-data-sgqlyj12", "time" : ISODate("2025-08-25T14:07:20.025Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0016"), "msg" : "kbcli-test-data-sgqlyj6", "time" : ISODate("2025-08-25T14:07:20.023Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a001a"), "msg" : "kbcli-test-data-sgqlyj10", "time" : ISODate("2025-08-25T14:07:20.025Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a0019"), "msg" : "kbcli-test-data-sgqlyj9", "time" : ISODate("2025-08-25T14:07:20.024Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a001b"), "msg" : "kbcli-test-data-sgqlyj11", "time" : ISODate("2025-08-25T14:07:20.025Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a001d"), "msg" : "kbcli-test-data-sgqlyj13", "time" : ISODate("2025-08-25T14:07:20.077Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a001f"), "msg" : "kbcli-test-data-sgqlyj15", "time" : ISODate("2025-08-25T14:07:20.078Z") *** *** "_id" : ObjectId("68ac6e18be1c41abe53a001e"), "msg" : "kbcli-test-data-sgqlyj14", "time" : ISODate("2025-08-25T14:07:20.078Z") *** bye dump backup check data Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-sgqlyj-backup` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-sgqlyj-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:h7B7m8Jz9786q3Ix;DB_PORT:27017;DB_DATABASE: No resources found in ns-thtcx namespace. `echo " echo \"rs.status()\" | mongo --host mongodb-sgqlyj-backup-mongodb-mongodb.ns-thtcx.svc.cluster.local --port 27017 -u root -p h7B7m8Jz9786q3Ix --authenticationDatabase admin admin " | kubectl exec -it mongodb-sgqlyj-backup-mongodb-0 --namespace ns-thtcx -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) Unable to use a TTY - input is not a terminal or the right kind of file Percona Server for MongoDB shell version v4.0.28-23 connecting to: mongodb://mongodb-sgqlyj-backup-mongodb-mongodb.ns-thtcx.svc.cluster.local:27017/admin?authSource=admin&gssapiServiceName=mongodb Implicit session: session *** "id" : UUID("7a533d0f-329a-460f-b476-aafc318b266e") *** Percona Server for MongoDB server version: v4.0.28-23 *** "set" : "mongodb-sgqlyj-backup-mongodb", "date" : ISODate("2025-08-25T14:09:56.479Z"), "myState" : 1, "term" : NumberLong(1), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "heartbeatIntervalMillis" : NumberLong(2000), "optimes" : *** "lastCommittedOpTime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "readConcernMajorityOpTime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "appliedOpTime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "durableOpTime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) *** ***, "lastStableCheckpointTimestamp" : Timestamp(1756130957, 5), "electionCandidateMetrics" : *** "lastElectionReason" : "electionTimeout", "lastElectionDate" : ISODate("2025-08-25T14:08:21.247Z"), "electionTerm" : NumberLong(1), "lastCommittedOpTimeAtElection" : *** "ts" : Timestamp(0, 0), "t" : NumberLong(-1) ***, "lastSeenOpTimeAtElection" : *** "ts" : Timestamp(1756130901, 1), "t" : NumberLong(-1) ***, "numVotesNeeded" : 1, "priorityAtElection" : 2, "electionTimeoutMillis" : NumberLong(10000), "newTermStartDate" : ISODate("2025-08-25T14:08:21.250Z"), "wMajorityWriteAvailabilityDate" : ISODate("2025-08-25T14:08:21.298Z") ***, "members" : [ *** "_id" : 0, "name" : "mongodb-sgqlyj-backup-mongodb-0.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "health" : 1, "state" : 1, "stateStr" : "PRIMARY", "uptime" : 102, "optime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "optimeDate" : ISODate("2025-08-25T14:09:52Z"), "syncingTo" : "", "syncSourceHost" : "", "syncSourceId" : -1, "infoMessage" : "could not find member to sync from", "electionTime" : Timestamp(1756130901, 2), "electionDate" : ISODate("2025-08-25T14:08:21Z"), "configVersion" : 3, "self" : true, "lastHeartbeatMessage" : "" ***, *** "_id" : 1, "name" : "mongodb-sgqlyj-backup-mongodb-1.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 49, "optime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "optimeDurable" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "optimeDate" : ISODate("2025-08-25T14:09:52Z"), "optimeDurableDate" : ISODate("2025-08-25T14:09:52Z"), "lastHeartbeat" : ISODate("2025-08-25T14:09:55.980Z"), "lastHeartbeatRecv" : ISODate("2025-08-25T14:09:54.983Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "mongodb-sgqlyj-backup-mongodb-0.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "syncSourceHost" : "mongodb-sgqlyj-backup-mongodb-0.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 3 ***, *** "_id" : 2, "name" : "mongodb-sgqlyj-backup-mongodb-2.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "health" : 1, "state" : 2, "stateStr" : "SECONDARY", "uptime" : 24, "optime" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "optimeDurable" : *** "ts" : Timestamp(1756130992, 6), "t" : NumberLong(1) ***, "optimeDate" : ISODate("2025-08-25T14:09:52Z"), "optimeDurableDate" : ISODate("2025-08-25T14:09:52Z"), "lastHeartbeat" : ISODate("2025-08-25T14:09:56.077Z"), "lastHeartbeatRecv" : ISODate("2025-08-25T14:09:55.076Z"), "pingMs" : NumberLong(0), "lastHeartbeatMessage" : "", "syncingTo" : "mongodb-sgqlyj-backup-mongodb-0.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "syncSourceHost" : "mongodb-sgqlyj-backup-mongodb-0.mongodb-sgqlyj-backup-mongodb-headless.ns-thtcx.svc:27017", "syncSourceId" : 0, "infoMessage" : "", "configVersion" : 3 *** ], "ok" : 1, "operationTime" : Timestamp(1756130992, 6), "$clusterTime" : *** "clusterTime" : Timestamp(1756130992, 6), "signature" : *** "hash" : BinData(0,"BTuiVD1AUwXuL2oWp90JdbhZSaA="), "keyId" : NumberLong("7542524787290013699") *** *** *** bye connect cluster Success delete cluster mongodb-sgqlyj-backup `kbcli cluster delete mongodb-sgqlyj-backup --auto-approve --namespace ns-thtcx ` Cluster mongodb-sgqlyj-backup deleted pod_info:mongodb-sgqlyj-backup-mongodb-0 4/4 Running 0 2m1s mongodb-sgqlyj-backup-mongodb-1 4/4 Running 0 96s mongodb-sgqlyj-backup-mongodb-2 4/4 Running 0 64s No resources found in ns-thtcx namespace. delete cluster pod done No resources found in ns-thtcx namespace. check cluster resource non-exist OK: pvc No resources found in ns-thtcx namespace. delete cluster done No resources found in ns-thtcx namespace. No resources found in ns-thtcx namespace. No resources found in ns-thtcx namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-thtcx-mongodb-sgqlyj-20250825220727 --namespace ns-thtcx ` backup.dataprotection.kubeblocks.io/backup-ns-thtcx-mongodb-sgqlyj-20250825220727 patched `kbcli cluster delete-backup mongodb-sgqlyj --name backup-ns-thtcx-mongodb-sgqlyj-20250825220727 --force --auto-approve --namespace ns-thtcx ` Backup backup-ns-thtcx-mongodb-sgqlyj-20250825220727 deleted No opsrequests found in ns-thtcx namespace. cluster list-logs `kbcli cluster list-logs mongodb-sgqlyj --namespace ns-thtcx ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update mongodb-sgqlyj --enable-all-logs=true --namespace ns-thtcx Error from server (NotFound): pods "mongodb-sgqlyj-mongodb-0" not found cluster logs `kbcli cluster logs mongodb-sgqlyj --tail 30 --namespace ns-thtcx ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, lorry, init-syncer (init), init-kubectl (init), init-pbm-agent (init) /src/highavailability/ha.go:89 github.com/apecloud/syncer/highavailability.(*Ha).Start /src/highavailability/ha.go:278 2025-08-25T13:59:40Z INFO MongoDB get replset status failed ***"error": "replSetGetStatus: connection(localhost:27017[-1100]) socket was unexpectedly closed: EOF"*** 2025-08-25T14:02:31Z INFO DCS-K8S lock expired: map[acquire-time:1756130272 available-time:1756130524 extra: holder:mongodb-sgqlyj-mongodb-2 leader:mongodb-sgqlyj-mongodb-2 renew-time:1756130524 ttl:15], now: 1756130551 2025-08-25T14:02:31Z INFO HA Cluster has no leader, attempt to take the leader 2025-08-25T14:02:31Z INFO HA there is a healthy leader exists ***"leader": "mongodb-sgqlyj-mongodb-2"*** 2025-08-25T14:02:57Z INFO MongoDB Get replSet config failed ***"error": "replSetGetConfig: connection() error occurred during connection handshake: auth error: sasl conversation error: unable to authenticate using mechanism \"SCRAM-SHA-256\": connection(mongodb-sgqlyj-mongodb-2.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017[-2266]) socket was unexpectedly closed: EOF"*** 2025-08-25T14:02:57Z INFO HA Cluster has no leader, attempt to take the leader 2025-08-25T14:02:57Z INFO HA there is a healthy leader exists ***"leader": "mongodb-sgqlyj-mongodb-2"*** 2025-08-25T14:02:58Z INFO HA Cluster has no leader, attempt to take the leader 2025-08-25T14:02:58Z INFO HA there is a healthy leader exists ***"leader": "mongodb-sgqlyj-mongodb-2"*** 2025-08-25T14:02:59Z INFO HA Cluster has no leader, attempt to take the leader 2025-08-25T14:02:59Z INFO HA there is a healthy leader exists ***"leader": "mongodb-sgqlyj-mongodb-1"*** 2025-08-25T14:04:42Z INFO DCS-K8S lock expired: map[acquire-time:1756130579 available-time:1756130666 extra: holder:mongodb-sgqlyj-mongodb-1 leader:mongodb-sgqlyj-mongodb-1 renew-time:1756130666 ttl:15], now: 1756130682 2025-08-25T14:04:42Z INFO HA Cluster has no leader, attempt to take the leader 2025-08-25T14:04:42Z INFO HA Failover happens, i am the healthiest member, attempt to take the leader 2025-08-25T14:04:42Z INFO event send event: map[member:mongodb-sgqlyj-mongodb-0 message:Failover happens, i am the healthiest member, attempt to take the leader operation:healthyCheck] 2025-08-25T14:04:42Z INFO MongoDB reconfig replset ***"config": ***"_id":"mongodb-sgqlyj-mongodb","version":17,"members":[***"_id":1,"host":"mongodb-sgqlyj-mongodb-1.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"slaveDelay":0,"votes":1***,***"_id":2,"host":"mongodb-sgqlyj-mongodb-2.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":1,"slaveDelay":0,"votes":1***,***"_id":3,"host":"mongodb-sgqlyj-mongodb-0.mongodb-sgqlyj-mongodb-headless.ns-thtcx.svc:27017","arbiterOnly":false,"buildIndexes":true,"hidden":false,"priority":2,"slaveDelay":0,"votes":1***],"protocolVersion":1,"settings":***"getLastErrorDefaults":***"w":null,"wtimeout":0***,"replicaSetId":"000000000000000000000000"***,"writeConcernMajorityJournalDefault":true*** 2025-08-25T14:05:13Z INFO HA Take the leader success! 2025-08-25T14:06:07Z INFO HA This member is Cluster's leader 2025-08-25T14:06:07Z DEBUG HA Refresh leader ttl 2025-08-25T14:07:07Z INFO HA This member is Cluster's leader 2025-08-25T14:07:07Z DEBUG HA Refresh leader ttl 2025-08-25T14:08:07Z INFO HA This member is Cluster's leader 2025-08-25T14:08:07Z DEBUG HA Refresh leader ttl 2025-08-25T14:09:07Z INFO HA This member is Cluster's leader 2025-08-25T14:09:07Z DEBUG HA Refresh leader ttl 2025-08-25T14:10:07Z INFO HA This member is Cluster's leader 2025-08-25T14:10:07Z DEBUG HA Refresh leader ttl cluster logs running `kbcli cluster logs mongodb-sgqlyj --tail 30 --file-type=running --namespace ns-thtcx ` error: get pod component definition name in cluster.yaml fail delete cluster mongodb-sgqlyj `kbcli cluster delete mongodb-sgqlyj --auto-approve --namespace ns-thtcx ` Cluster mongodb-sgqlyj deleted pod_info:mongodb-sgqlyj-mongodb-0 4/4 Running 0 15m mongodb-sgqlyj-mongodb-1 4/4 Running 0 14m mongodb-sgqlyj-mongodb-2 4/4 Running 0 7m17s No resources found in ns-thtcx namespace. delete cluster pod done No resources found in ns-thtcx namespace. check cluster resource non-exist OK: pvc No resources found in ns-thtcx namespace. delete cluster done No resources found in ns-thtcx namespace. No resources found in ns-thtcx namespace. No resources found in ns-thtcx namespace. Mongodb Test Suite All Done! --------------------------------------Mongodb (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentVersion=mongodb;ServiceVersion=4.0.28;]|[Description=Create a cluster and component version mongodb and service version 4.0.28] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=feqkr]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=feqkr;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=mongodb]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=mongodb]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mongodb]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Failover]|[HA=Pod Kill;ComponentName=mongodb]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [PASSED]|[RebuildInstance]|[ComponentName=mongodb]|[Description=Rebuild the cluster instance specify component mongodb] [PASSED]|[HorizontalScaling Out]|[ComponentName=mongodb]|[Description=HorizontalScaling Out the cluster specify component mongodb] [PASSED]|[HorizontalScaling In]|[ComponentName=mongodb]|[Description=HorizontalScaling In the cluster specify component mongodb] [WARNING]|[CheckFailover]|[FailoverType=timeoffset]|[Description=-] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=mongodb]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[VolumeExpansion]|[ComponentName=mongodb]|[Description=VolumeExpansion the cluster specify component mongodb] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Reconfiguring]|[ComponentName=mongodb;systemLog.quiet=false,systemLog.verbosity=1]|[Description=Reconfiguring the cluster specify component mongodb set systemLog.quiet=false,systemLog.verbosity=1] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=mongodb]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[SwitchOver]|[ComponentName=mongodb]|[Description=SwitchOver the cluster specify component mongodb] [PASSED]|[Reconfiguring]|[ComponentName=mongodb;systemLog.quiet=true]|[Description=Reconfiguring the cluster specify component mongodb set systemLog.quiet=true] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[VerticalScaling]|[ComponentName=mongodb]|[Description=VerticalScaling the cluster specify component mongodb] [PASSED]|[HscaleOfflineInstances]|[ComponentName=mongodb]|[Description=Hscale the cluster instances offline specify component mongodb] [PASSED]|[HscaleOnlineInstances]|[ComponentName=mongodb]|[Description=Hscale the cluster instances online specify component mongodb] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=mongodb]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=mongodb]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=mongodb]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=mongodb]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=dump]|[Description=The cluster dump Backup] [PASSED]|[Restore]|[BackupMethod=dump]|[Description=The cluster dump Restore] [PASSED]|[Check Data]|[BackupMethod=dump]|[Description=Check the cluster data restore via dump] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=dump]|[Description=Delete the dump restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]