source commons files source engines files source kubeblocks files source kubedb files CLUSTER_NAME: `kubectl get namespace | grep ns-vuvkk ` `kubectl create namespace ns-vuvkk` namespace/ns-vuvkk created create namespace ns-vuvkk done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 83.4M 0 --:--:-- --:--:-- --:--:-- 83.4M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.1 done Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Kubernetes Env: v1.32.6 check snapshot controller check snapshot controller done POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default KubeBlocks version is:1.0.1 skip upgrade KubeBlocks current KubeBlocks version: 1.0.1 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:mongodb set component version set component version:mongodb set service versions:8.0.8,7.0.18,6.0.21,5.0.29,4.4.29 set service versions sorted:4.4.29,5.0.29,6.0.21,7.0.18,8.0.8 set mongodb component definition set mongodb component definition mongodb-1.0.1 REPORT_COUNT 0:0 set replicas first:3,4.4.29|3,5.0.29|3,6.0.21|3,7.0.18|3,8.0.8 set replicas third:3,7.0.18 set replicas fourth:3,7.0.18 set minimum cmpv service version set minimum cmpv service version replicas:3,7.0.18 REPORT_COUNT:1 CLUSTER_TOPOLOGY:replicaset topology replicaset found in cluster definition mongodb set mongodb component definition set mongodb component definition mongodb-1.0.1 LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 3 CLUSTER_NAME:mongodb-ejjbon No resources found in ns-vuvkk namespace. pod_info: termination_policy:DoNotTerminate create 3 replica DoNotTerminate mongodb cluster check component definition set component definition by component version check cmpd by labels check cmpd by compDefs set component definition: mongodb-1.0.1 by component version:mongodb apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: mongodb-ejjbon namespace: ns-vuvkk spec: clusterDef: mongodb topology: replicaset terminationPolicy: DoNotTerminate componentSpecs: - name: mongodb serviceVersion: 7.0.18 replicas: 3 resources: limits: cpu: 100m memory: 0.5Gi requests: cpu: 100m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi `kubectl apply -f test_create_mongodb-ejjbon.yaml` cluster.apps.kubeblocks.io/mongodb-ejjbon created apply test_create_mongodb-ejjbon.yaml Success `rm -rf test_create_mongodb-ejjbon.yaml` check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Creating Sep 11,2025 17:21 UTC+0800 clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:23 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:24 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: check pod mongodb-ejjbon-mongodb-0 container_name mongodb exist password 16yFD9v3AK34W9u1 Container mongodb logs contain secret password:2025-09-11T09:23:09Z INFO MongoDB Create user: root, passwd: 16yFD9v3AK34W9u1, roles: map[db:admin role:root] describe cluster `kbcli cluster describe mongodb-ejjbon --namespace ns-vuvkk ` Name: mongodb-ejjbon Created Time: Sep 11,2025 17:21 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-vuvkk mongodb replicaset Running DoNotTerminate Endpoints: COMPONENT INTERNAL EXTERNAL mongodb mongodb-ejjbon-mongodb.ns-vuvkk.svc.cluster.local:27017 mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017 mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME mongodb 7.0.18 mongodb-ejjbon-mongodb-0 primary Running 0 aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:21 UTC+0800 mongodb 7.0.18 mongodb-ejjbon-mongodb-1 secondary Running 0 aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:23 UTC+0800 mongodb 7.0.18 mongodb-ejjbon-mongodb-2 secondary Running 0 aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:24 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mongodb 100m / 100m 512Mi / 512Mi data:3Gi default Images: COMPONENT COMPONENT-DEFINITION IMAGE mongodb mongodb-1.0.1 docker.io/apecloud/percona-server-mongodb:7.0.18 docker.io/apecloud/percona-backup-mongodb:2.9.1 docker.io/apecloud/mongodb_exporter:0.44.0 docker.io/apecloud/kubeblocks-tools:1.0.1 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-vuvkk mongodb-ejjbon `kbcli cluster label mongodb-ejjbon app.kubernetes.io/instance- --namespace ns-vuvkk ` label "app.kubernetes.io/instance" not found. `kbcli cluster label mongodb-ejjbon app.kubernetes.io/instance=mongodb-ejjbon --namespace ns-vuvkk ` `kbcli cluster label mongodb-ejjbon --list --namespace ns-vuvkk ` NAME NAMESPACE LABELS mongodb-ejjbon ns-vuvkk app.kubernetes.io/instance=mongodb-ejjbon clusterdefinition.kubeblocks.io/name=mongodb label cluster app.kubernetes.io/instance=mongodb-ejjbon Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=mongodb-ejjbon --namespace ns-vuvkk ` `kbcli cluster label mongodb-ejjbon --list --namespace ns-vuvkk ` NAME NAMESPACE LABELS mongodb-ejjbon ns-vuvkk app.kubernetes.io/instance=mongodb-ejjbon case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=mongodb label cluster case.name=kbcli.test1 Success `kbcli cluster label mongodb-ejjbon case.name=kbcli.test2 --overwrite --namespace ns-vuvkk ` `kbcli cluster label mongodb-ejjbon --list --namespace ns-vuvkk ` NAME NAMESPACE LABELS mongodb-ejjbon ns-vuvkk app.kubernetes.io/instance=mongodb-ejjbon case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=mongodb label cluster case.name=kbcli.test2 Success `kbcli cluster label mongodb-ejjbon case.name- --namespace ns-vuvkk ` `kbcli cluster label mongodb-ejjbon --list --namespace ns-vuvkk ` NAME NAMESPACE LABELS mongodb-ejjbon ns-vuvkk app.kubernetes.io/instance=mongodb-ejjbon clusterdefinition.kubeblocks.io/name=mongodb delete cluster label case.name Success list-accounts on characterType mongodb is not supported yet cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo " echo \"rs.status()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c295b7e7ef43591ba762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:22:55.502+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:23:01.935+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:23:01.935+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:23:01.935+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> *** set: 'mongodb-ejjbon-mongodb', date: ISODate('2025-09-11T09:26:29.895Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1757582789, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-09-11T09:26:29.693Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1757582789, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1757582789, i: 2 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1757582789, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-09-11T09:26:29.894Z'), lastDurableWallTime: ISODate('2025-09-11T09:26:29.693Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1757582761, i: 2 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-09-11T09:23:04.012Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1757582583, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1757582583, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 1, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), newTermStartDate: ISODate('2025-09-11T09:23:04.293Z'), wMajorityWriteAvailabilityDate: ISODate('2025-09-11T09:23:04.395Z') ***, members: [ *** _id: 0, name: 'mongodb-ejjbon-mongodb-0.mongodb-ejjbon-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 214, optime: *** ts: Timestamp(*** t: 1757582789, i: 2 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T09:26:29.000Z'), lastAppliedWallTime: ISODate('2025-09-11T09:26:29.894Z'), lastDurableWallTime: ISODate('2025-09-11T09:26:29.693Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp(*** t: 1757582584, i: 1 ***), electionDate: ISODate('2025-09-11T09:23:04.000Z'), configVersion: 5, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-ejjbon-mongodb-1.mongodb-ejjbon-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 136, optime: *** ts: Timestamp(*** t: 1757582786, i: 2 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757582786, i: 2 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T09:26:26.000Z'), optimeDurableDate: ISODate('2025-09-11T09:26:26.000Z'), lastAppliedWallTime: ISODate('2025-09-11T09:26:29.693Z'), lastDurableWallTime: ISODate('2025-09-11T09:26:29.693Z'), lastHeartbeat: ISODate('2025-09-11T09:26:29.791Z'), lastHeartbeatRecv: ISODate('2025-09-11T09:26:29.692Z'), pingMs: Long('47'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-mongodb-0.mongodb-ejjbon-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 5, configTerm: 1 ***, *** _id: 2, name: 'mongodb-ejjbon-mongodb-2.mongodb-ejjbon-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 82, optime: *** ts: Timestamp(*** t: 1757582786, i: 2 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757582786, i: 2 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T09:26:26.000Z'), optimeDurableDate: ISODate('2025-09-11T09:26:26.000Z'), lastAppliedWallTime: ISODate('2025-09-11T09:26:29.693Z'), lastDurableWallTime: ISODate('2025-09-11T09:26:29.693Z'), lastHeartbeat: ISODate('2025-09-11T09:26:29.692Z'), lastHeartbeatRecv: ISODate('2025-09-11T09:26:29.793Z'), pingMs: Long('69'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-mongodb-1.mongodb-ejjbon-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 1, infoMessage: '', configVersion: 5, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1757582789, i: 2 ***), signature: *** hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0), keyId: Long('0') *** ***, operationTime: Timestamp(*** t: 1757582789, i: 2 ***) *** mongodb-ejjbon-mongodb [direct: primary] admin> connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-mongodb-ejjbon" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mongodb-ejjbon --namespace ns-vuvkk ` Error from server (NotFound): pods "test-db-client-executionloop-mongodb-ejjbon" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-mongodb-ejjbon" not found `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-mongodb-ejjbon namespace: ns-vuvkk spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mongodb-ejjbon-mongodb.ns-vuvkk.svc.cluster.local" - "--user" - "root" - "--password" - "16yFD9v3AK34W9u1" - "--port" - "27017" - "--dbtype" - "mongodb" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-mongodb-ejjbon.yaml` pod/test-db-client-executionloop-mongodb-ejjbon created apply test-db-client-executionloop-mongodb-ejjbon.yaml Success `rm -rf test-db-client-executionloop-mongodb-ejjbon.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 9s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 14s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 1/1 Running 0 62s check pod test-db-client-executionloop-mongodb-ejjbon status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-ejjbon 0/1 Completed 0 67s check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:23 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:24 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done Inserted document: BsonObjectId***value=68c29601d65c3f63f7346980*** Inserted document: BsonObjectId***value=68c29601d65c3f63f7346981*** Inserted document: BsonObjectId***value=68c29601d65c3f63f7346982*** [ 51s ] executions total: 587 successful: 585 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29601d65c3f63f7346983*** Inserted document: BsonObjectId***value=68c29601d65c3f63f7346984*** Inserted document: BsonObjectId***value=68c29602d65c3f63f7346985*** Inserted document: BsonObjectId***value=68c29602d65c3f63f7346986*** Inserted document: BsonObjectId***value=68c29602d65c3f63f7346987*** Inserted document: BsonObjectId***value=68c29602d65c3f63f7346988*** Inserted document: BsonObjectId***value=68c29602d65c3f63f7346989*** Inserted document: BsonObjectId***value=68c29602d65c3f63f734698a*** Inserted document: BsonObjectId***value=68c29602d65c3f63f734698b*** Inserted document: BsonObjectId***value=68c29602d65c3f63f734698c*** Inserted document: BsonObjectId***value=68c29602d65c3f63f734698d*** [ 52s ] executions total: 598 successful: 596 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29602d65c3f63f734698e*** Inserted document: BsonObjectId***value=68c29603d65c3f63f734698f*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346990*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346991*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346992*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346993*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346994*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346995*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346996*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346997*** Inserted document: BsonObjectId***value=68c29603d65c3f63f7346998*** [ 53s ] executions total: 609 successful: 607 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29603d65c3f63f7346999*** Inserted document: BsonObjectId***value=68c29604d65c3f63f734699a*** Inserted document: BsonObjectId***value=68c29604d65c3f63f734699b*** Inserted document: BsonObjectId***value=68c29604d65c3f63f734699c*** Inserted document: BsonObjectId***value=68c29604d65c3f63f734699d*** Inserted document: BsonObjectId***value=68c29604d65c3f63f734699e*** Inserted document: BsonObjectId***value=68c29604d65c3f63f734699f*** Inserted document: BsonObjectId***value=68c29604d65c3f63f73469a0*** Inserted document: BsonObjectId***value=68c29604d65c3f63f73469a1*** Inserted document: BsonObjectId***value=68c29604d65c3f63f73469a2*** Inserted document: BsonObjectId***value=68c29604d65c3f63f73469a3*** Inserted document: BsonObjectId***value=68c29604d65c3f63f73469a4*** [ 54s ] executions total: 621 successful: 619 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29605d65c3f63f73469a5*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469a6*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469a7*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469a8*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469a9*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469aa*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469ab*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469ac*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469ad*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469ae*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469af*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469b0*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469b1*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469b2*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469b3*** Inserted document: BsonObjectId***value=68c29605d65c3f63f73469b4*** [ 55s ] executions total: 637 successful: 635 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29606d65c3f63f73469b5*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469b6*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469b7*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469b8*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469b9*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469ba*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469bb*** Inserted document: BsonObjectId***value=68c29606d65c3f63f73469bc*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469bd*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469be*** [ 56s ] executions total: 647 successful: 645 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29607d65c3f63f73469bf*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c0*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c1*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c2*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c3*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c4*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c5*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c6*** Inserted document: BsonObjectId***value=68c29607d65c3f63f73469c7*** [ 57s ] executions total: 656 successful: 654 failed: 2 disconnect: 1 Inserted document: BsonObjectId***value=68c29608d65c3f63f73469c8*** [ 60s ] executions total: 657 successful: 655 failed: 2 disconnect: 1 Test Result: Total Executions: 657 Successful Executions: 655 Failed Executions: 2 Disconnection Counts: 1 Connection Information: Database Type: mongodb Host: mongodb-ejjbon-mongodb.ns-vuvkk.svc.cluster.local Port: 27017 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 655 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mongodb-ejjbon --namespace ns-vuvkk ` pod/test-db-client-executionloop-mongodb-ejjbon patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-mongodb-ejjbon" force deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.remove(***) ; db.col.insertOne(***a:'ozxfv'***)\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c296339af82c94a0a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:22:55.502+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:23:01.935+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:23:01.935+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:23:01.935+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> DeprecationWarning: Collection.remove() is deprecated. Use deleteOne, deleteMany, findOneAndDelete, or bulkWrite. *** acknowledged: true, insertedId: ObjectId('68c296429af82c94a0a762f9') *** mongodb-ejjbon-mongodb [direct: primary] admin> add consistent data ozxfv Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.remove(***) ; db.col.insertOne(***a:'ozxfv'***)\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2965854ecc5f96ba762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T09:24:59.327+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:25:05.841+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:25:05.841+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:25:05.841+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> DeprecationWarning: Collection.remove() is deprecated. Use deleteOne, deleteMany, findOneAndDelete, or bulkWrite. Uncaught MongoServerError[NotWritablePrimary]: not primary mongodb-ejjbon-mongodb [direct: secondary] admin> check add consistent data readonly Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart mongodb-ejjbon --auto-approve --force=true --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-restart-6jr2s created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-restart-6jr2s -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-restart-6jr2s ns-vuvkk Restart mongodb-ejjbon mongodb Running 0/3 Sep 11,2025 17:29 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-restart-6jr2s ns-vuvkk Restart mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 17:29 UTC+0800 check ops status done ops_status:mongodb-ejjbon-restart-6jr2s ns-vuvkk Restart mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 17:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-restart-6jr2s --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-restart-6jr2s patched `kbcli cluster delete-ops --name mongodb-ejjbon-restart-6jr2s --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-restart-6jr2s deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29706109f610257a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29722d715941d7ea762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T09:30:54.101+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:31:04.604+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:31:04.604+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:31:04.604+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-ejjbon" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-2 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-mongodb-ejjbon.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-mongodb-ejjbon created apply test-chaos-mesh-dnserror-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-dnserror-mongodb-ejjbon.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-ejjbon" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-2 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c297f6a9389f5122a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29811fb58da485ea762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:30:54.101+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:31:04.604+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:31:04.604+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:31:04.604+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-ejjbon" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-2 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-mongodb-ejjbon.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-mongodb-ejjbon created apply test-chaos-mesh-networklossover-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-networklossover-mongodb-ejjbon.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-0 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-ejjbon" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-1 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c298da503f24a66fa762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:30:19.420+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:30:29.523+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:30:29.523+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:30:29.523+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c298f3bce586a8b6a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-ejjbon" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-1 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-mongodb-ejjbon.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-mongodb-ejjbon created apply test-chaos-mesh-podfailure-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-podfailure-mongodb-ejjbon.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Failed Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role primary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 primary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 primary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-ejjbon" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c299caa0a5ca23bda762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:30:54.101+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:31:04.604+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:31:04.604+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:31:04.604+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c299e3189a6e5ac9a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-ejjbon" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-mongodb-ejjbon.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mongodb-ejjbon created apply test-chaos-mesh-networkcorruptover-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-mongodb-ejjbon.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role primary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 primary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 primary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 check cluster role done primary: mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-ejjbon" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-1 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29ab35efade4fc5a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:42:49.918+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:42:59.924+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:42:59.924+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:42:59.924+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29acd9420ea28a5a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:30:54.101+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:31:04.604+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:31:04.604+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:31:04.604+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-ejjbon" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-1 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mongodb-ejjbon.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mongodb-ejjbon created apply test-chaos-mesh-oom-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-oom-mongodb-ejjbon.yaml` check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-ejjbon" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mongodb-ejjbon patched check failover pod name failover pod name:mongodb-ejjbon-mongodb-2 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29b3dc6ad358e66a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29b579c74eb38a4a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:49:02.419+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:49:12.423+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:49:12.423+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:49:12.423+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster mongodb scale-out cluster mongodb scale-out replicas: 4 check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-ejjbon namespace. `kbcli cluster scale-out mongodb-ejjbon --auto-approve --force=true --components mongodb --replicas 1 --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-horizontalscaling-mrx88 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-horizontalscaling-mrx88 -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-horizontalscaling-mrx88 ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 17:50 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 mongodb-ejjbon-mongodb-3 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:50 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-3 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done No resources found in mongodb-ejjbon namespace. check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-horizontalscaling-mrx88 ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 17:50 UTC+0800 check ops status done ops_status:mongodb-ejjbon-horizontalscaling-mrx88 ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 17:50 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-horizontalscaling-mrx88 --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-horizontalscaling-mrx88 patched `kbcli cluster delete-ops --name mongodb-ejjbon-horizontalscaling-mrx88 --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-horizontalscaling-mrx88 deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29bf8fbc30505d3a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29c140a306b45f6a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:49:02.419+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:49:12.423+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:49:12.423+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:49:12.423+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster mongodb scale-in cluster mongodb scale-in replicas: 3 check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-ejjbon namespace. `kbcli cluster scale-in mongodb-ejjbon --auto-approve --force=true --components mongodb --replicas 1 --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-horizontalscaling-w86pv created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-horizontalscaling-w86pv -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-horizontalscaling-w86pv ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 17:54 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:3Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done No resources found in mongodb-ejjbon namespace. check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-horizontalscaling-w86pv ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 17:54 UTC+0800 check ops status done ops_status:mongodb-ejjbon-horizontalscaling-w86pv ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 17:54 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-horizontalscaling-w86pv --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-horizontalscaling-w86pv patched `kbcli cluster delete-ops --name mongodb-ejjbon-horizontalscaling-w86pv --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-horizontalscaling-w86pv deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29c82e7cd21d50da762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29c9bdc698edb4ea762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:49:02.419+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:49:12.423+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:49:12.423+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:49:12.423+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `mongosh ***mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg:Current Mongosh Log ID: 68c29d0b454c8761b2a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local/?directConnection=true&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] test> connect headlessEndpoints Success `kubectl get pvc -l app.kubernetes.io/instance=mongodb-ejjbon,apps.kubeblocks.io/component-name=mongodb,apps.kubeblocks.io/vct-name=data --namespace ns-vuvkk ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-ejjbon namespace. `kbcli cluster volume-expand mongodb-ejjbon --auto-approve --force=true --components mongodb --volume-claim-templates data --storage 4Gi --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-volumeexpansion-wrm9n created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-volumeexpansion-wrm9n -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-volumeexpansion-wrm9n ns-vuvkk VolumeExpansion mongodb-ejjbon mongodb Running 0/3 Sep 11,2025 17:57 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done No resources found in mongodb-ejjbon namespace. check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-volumeexpansion-wrm9n ns-vuvkk VolumeExpansion mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 17:57 UTC+0800 check ops status done ops_status:mongodb-ejjbon-volumeexpansion-wrm9n ns-vuvkk VolumeExpansion mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 17:57 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-volumeexpansion-wrm9n --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-volumeexpansion-wrm9n patched `kbcli cluster delete-ops --name mongodb-ejjbon-volumeexpansion-wrm9n --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-volumeexpansion-wrm9n deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29ec1c1ddec0289a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29edcb5cd2d2b1ea762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:30:54.101+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:31:04.604+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:31:04.604+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:31:04.604+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-ejjbon" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-2 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-mongodb-ejjbon.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-mongodb-ejjbon created apply test-chaos-mesh-dnsrandom-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-dnsrandom-mongodb-ejjbon.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 100m / 100m 512Mi / 512Mi data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 17:30 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 100m / 100m 512Mi / 512Mi data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 17:29 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-ejjbon" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-2 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29fa8fc9c7fd166a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:29:40.426+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:29:50.633+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:29:50.633+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:29:50.633+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c29fc4d8db3d78c2a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T09:30:54.101+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T09:31:04.604+00:00: You are running this process as the root user, which is not recommended 2025-09-11T09:31:04.604+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T09:31:04.604+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale mongodb-ejjbon --auto-approve --force=true --components mongodb --cpu 200m --memory 0.6Gi --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-verticalscaling-d22jm created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-verticalscaling-d22jm -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-verticalscaling-d22jm ns-vuvkk VerticalScaling mongodb-ejjbon mongodb Running 0/3 Sep 11,2025 18:09 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:12 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:10 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:11 UTC+0800 check pod status done check cluster role No resources found in ns-vuvkk namespace. primary: ;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 No resources found in ns-vuvkk namespace. primary: ;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 No resources found in ns-vuvkk namespace. primary: ;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 check cluster role done primary: mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-verticalscaling-d22jm ns-vuvkk VerticalScaling mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 18:09 UTC+0800 check ops status done ops_status:mongodb-ejjbon-verticalscaling-d22jm ns-vuvkk VerticalScaling mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 18:09 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-verticalscaling-d22jm --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-verticalscaling-d22jm patched `kbcli cluster delete-ops --name mongodb-ejjbon-verticalscaling-d22jm --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-verticalscaling-d22jm deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a0a62e2cba3f9fa762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:11:21.538+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:11:26.542+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:11:26.542+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:11:26.542+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a0b16320fb5fefa762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:10:36.928+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:10:42.129+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:10:42.129+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:10:42.129+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster does not need to check monitor currently check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:12 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:10 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:11 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-ejjbon-rebuildinstance- namespace: ns-vuvkk spec: type: RebuildInstance clusterName: mongodb-ejjbon force: true rebuildFrom: - componentName: mongodb instances: - name: mongodb-ejjbon-mongodb-0 inPlace: true check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-ejjbon.yaml` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-rebuildinstance-fzzml created create test_ops_cluster_mongodb-ejjbon.yaml Success `rm -rf test_ops_cluster_mongodb-ejjbon.yaml` check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:13 UTC+0800 check ops status done ops_status:mongodb-ejjbon-rebuildinstance-fzzml ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 18:13 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-rebuildinstance-fzzml --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-rebuildinstance-fzzml patched `kbcli cluster delete-ops --name mongodb-ejjbon-rebuildinstance-fzzml --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-rebuildinstance-fzzml deleted check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:14 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:10 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:11 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a1495e1d59f640a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:11:21.538+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:11:26.542+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:11:26.542+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:11:26.542+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a1531f0fc79b4ca762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:10:36.928+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:10:42.129+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:10:42.129+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:10:42.129+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster hscale offline instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-ejjbon-hscaleoffinstance- labels: app.kubernetes.io/instance: mongodb-ejjbon app.kubernetes.io/managed-by: kubeblocks namespace: ns-vuvkk spec: type: HorizontalScaling clusterName: mongodb-ejjbon force: true horizontalScaling: - componentName: mongodb scaleIn: onlineInstancesToOffline: - mongodb-ejjbon-mongodb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-ejjbon.yaml` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-hscaleoffinstance-f5q6j created create test_ops_cluster_mongodb-ejjbon.yaml Success `rm -rf test_ops_cluster_mongodb-ejjbon.yaml` check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-hscaleoffinstance-f5q6j ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:16 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:10 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:11 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-hscaleoffinstance-f5q6j ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 18:16 UTC+0800 check ops status done ops_status:mongodb-ejjbon-hscaleoffinstance-f5q6j ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 18:16 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-hscaleoffinstance-f5q6j --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-hscaleoffinstance-f5q6j patched `kbcli cluster delete-ops --name mongodb-ejjbon-hscaleoffinstance-f5q6j --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-hscaleoffinstance-f5q6j deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a18c51a82a083aa762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:11:21.538+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:11:26.542+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:11:26.542+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:11:26.542+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a195ea806c7a91a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:10:36.928+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:10:42.129+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:10:42.129+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:10:42.129+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster hscale online instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-ejjbon-hscaleoninstance- labels: app.kubernetes.io/instance: mongodb-ejjbon app.kubernetes.io/managed-by: kubeblocks namespace: ns-vuvkk spec: type: HorizontalScaling clusterName: mongodb-ejjbon force: true horizontalScaling: - componentName: mongodb scaleOut: offlineInstancesToOnline: - mongodb-ejjbon-mongodb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-ejjbon.yaml` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-hscaleoninstance-nfcvh created create test_ops_cluster_mongodb-ejjbon.yaml Success `rm -rf test_ops_cluster_mongodb-ejjbon.yaml` check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-hscaleoninstance-nfcvh ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:17 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:17 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:10 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:11 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-hscaleoninstance-nfcvh ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 18:17 UTC+0800 check ops status done ops_status:mongodb-ejjbon-hscaleoninstance-nfcvh ns-vuvkk HorizontalScaling mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 18:17 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-hscaleoninstance-nfcvh --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-hscaleoninstance-nfcvh patched `kbcli cluster delete-ops --name mongodb-ejjbon-hscaleoninstance-nfcvh --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-hscaleoninstance-nfcvh deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a2558d4d03f6dfa762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:11:21.538+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:11:26.542+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:11:26.542+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:11:26.542+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a25ffd0a3ee2a3a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:17:30.393+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:17:33.330+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:17:33.330+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:17:33.330+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop mongodb-ejjbon --auto-approve --force=true --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-stop-46bg8 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-stop-46bg8 -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-stop-46bg8 ns-vuvkk Stop mongodb-ejjbon mongodb Running 0/3 Sep 11,2025 18:20 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Stopping Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-stop-46bg8 ns-vuvkk Stop mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 18:20 UTC+0800 check ops status done ops_status:mongodb-ejjbon-stop-46bg8 ns-vuvkk Stop mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 18:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-stop-46bg8 --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-stop-46bg8 patched `kbcli cluster delete-ops --name mongodb-ejjbon-stop-46bg8 --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-stop-46bg8 deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start mongodb-ejjbon --force=true --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-start-9r4mz created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-start-9r4mz -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-start-9r4mz ns-vuvkk Start mongodb-ejjbon mongodb Running 0/3 Sep 11,2025 18:21 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:22 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-start-9r4mz ns-vuvkk Start mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 18:21 UTC+0800 check ops status done ops_status:mongodb-ejjbon-start-9r4mz ns-vuvkk Start mongodb-ejjbon mongodb Succeed 3/3 Sep 11,2025 18:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-start-9r4mz --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-start-9r4mz patched `kbcli cluster delete-ops --name mongodb-ejjbon-start-9r4mz --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-start-9r4mz deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a35e084452f1b6a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:22:51.497+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:22:56.499+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:22:56.499+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:22:56.499+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a368de473f2f08a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-ejjbon" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-mongodb-ejjbon.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-mongodb-ejjbon created apply test-chaos-mesh-timeoffset-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-timeoffset-mongodb-ejjbon.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:22 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-ejjbon" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a4073126674bafa762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:22:51.497+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:22:56.499+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:22:56.499+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:22:56.499+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a411cc1051f7a5a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-ejjbon" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-mongodb-ejjbon.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-mongodb-ejjbon created apply test-chaos-mesh-networkduplicate-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-networkduplicate-mongodb-ejjbon.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:22 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-ejjbon" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a4ad249d3cb94ba762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:22:51.497+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:22:56.499+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:22:56.499+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:22:56.499+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a4b7998296531ba762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote mongodb-ejjbon --auto-approve --force=true --instance mongodb-ejjbon-mongodb-0 --candidate mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-switchover-kqrjb created successfully, you can view the progress: kbcli cluster describe-ops mongodb-ejjbon-switchover-kqrjb -n ns-vuvkk check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-switchover-kqrjb ns-vuvkk Switchover mongodb-ejjbon mongodb-ejjbon-mongodb Running 0/1 Sep 11,2025 18:30 UTC+0800 check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:22 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-switchover-kqrjb ns-vuvkk Switchover mongodb-ejjbon mongodb-ejjbon-mongodb Succeed 1/1 Sep 11,2025 18:30 UTC+0800 check ops status done ops_status:mongodb-ejjbon-switchover-kqrjb ns-vuvkk Switchover mongodb-ejjbon mongodb-ejjbon-mongodb Succeed 1/1 Sep 11,2025 18:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-switchover-kqrjb --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-switchover-kqrjb patched `kbcli cluster delete-ops --name mongodb-ejjbon-switchover-kqrjb --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-switchover-kqrjb deleted `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a4e88c044afcdca762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:16.130+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:21.233+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:21.233+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:21.233+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a4f2a57a7ac1e6a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check db_client batch data Success switchover pod:mongodb-ejjbon-mongodb-1 switchover success test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-mongodb-ejjbon" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-1 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-mongodb-ejjbon.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-mongodb-ejjbon created apply test-chaos-mesh-podkill-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-podkill-mongodb-ejjbon.yaml` check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Updating Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-mongodb-ejjbon" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-mongodb-ejjbon patched check failover pod name failover pod name:mongodb-ejjbon-mongodb-2 failover podkill Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a554dbb2f54501a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a55f153776f294a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:32:08.926+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:32:13.331+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:32:13.331+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:32:13.332+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-ejjbon" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-2 action: partition mode: all target: mode: all selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-mongodb-ejjbon.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-mongodb-ejjbon created apply test-chaos-mesh-networkpartition-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-networkpartition-mongodb-ejjbon.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-0;secondary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-ejjbon" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-0 failover networkpartition Failure `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a5fc7255fdded7a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:22:51.497+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:22:56.499+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:22:56.499+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:22:56.499+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a606dd5a57004ba762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-ejjbon" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-mongodb-ejjbon.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-mongodb-ejjbon created apply test-chaos-mesh-networkbandwidthover-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-mongodb-ejjbon.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-ejjbon" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-1 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a6a3b7141d351da762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:32:08.926+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:32:13.331+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:32:13.331+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:32:13.332+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a6adbd3da1b116a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check db_client batch data Success skip cluster Upgrade test failover fullcpu check cluster status before cluster-failover-fullcpu check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-ejjbon" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-mongodb-ejjbon.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-mongodb-ejjbon created apply test-chaos-mesh-fullcpu-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-fullcpu-mongodb-ejjbon.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-ejjbon" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-1 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a748b0a279fe4ba762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:32:08.926+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:32:13.331+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:32:13.331+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:32:13.332+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a752da07bfc893a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-ejjbon" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-ejjbon" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-mongodb-ejjbon namespace: ns-vuvkk spec: selector: namespaces: - ns-vuvkk labelSelectors: apps.kubeblocks.io/pod-name: mongodb-ejjbon-mongodb-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-mongodb-ejjbon.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-mongodb-ejjbon created apply test-chaos-mesh-networkdelay-mongodb-ejjbon.yaml Success `rm -rf test-chaos-mesh-networkdelay-mongodb-ejjbon.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-1;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mongodb-ejjbon --namespace ns-vuvkk ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-ejjbon" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-ejjbon" not found check failover pod name failover pod name:mongodb-ejjbon-mongodb-1 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a7eee8402c2062a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:32:08.926+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:32:13.331+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:32:13.331+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:32:13.332+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a7f8cfb977ab07a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-1 --namespace ns-vuvkk -- bash ` check db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running `kill 1` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb DoNotTerminate Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 primary: mongodb-ejjbon-mongodb-1 mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done check failover pod name failover pod name:mongodb-ejjbon-mongodb-2 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a832b5ee8bc8c1a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a83ce6f4a792b9a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:44:42.018+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:44:47.223+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:44:47.223+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:44:47.224+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cmpv upgrade service version:3,4.4.29|3,5.0.29|3,6.0.21|3,7.0.18|3,8.0.8 set latest cmpv service version latest service version:7.0.18 cmpv service version upgrade and downgrade cluster update terminationPolicy WipeOut `kbcli cluster update mongodb-ejjbon --termination-policy=WipeOut --namespace ns-vuvkk ` cluster.apps.kubeblocks.io/mongodb-ejjbon updated check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb WipeOut Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:21 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done cluster datafile backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-fc6xr -n kb-heauh -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-fc6xr -n kb-heauh -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-heauh get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-heauh -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-heauh -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-heauh.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-heauh.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-8f45f86b6-jvj74 --namespace kb-heauh -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mongodb-ejjbon --method datafile --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911184553 created successfully, you can view the progress: kbcli cluster list-backups --names=backup-ns-vuvkk-mongodb-ejjbon-20250911184553 -n ns-vuvkk check backup status `kbcli cluster list-backups mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-vuvkk-mongodb-ejjbon-20250911184553 ns-vuvkk mongodb-ejjbon datafile Running Delete Sep 11,2025 18:45 UTC+0800 backup_status:mongodb-ejjbon-datafile-Running backup_status:mongodb-ejjbon-datafile-Running backup_status:mongodb-ejjbon-datafile-Running backup_status:mongodb-ejjbon-datafile-Running backup_status:mongodb-ejjbon-datafile-Running check backup status done backup_status:backup-ns-vuvkk-mongodb-ejjbon-20250911184553 ns-vuvkk mongodb-ejjbon datafile Completed 1984196 23s Delete Sep 11,2025 18:45 UTC+0800 Sep 11,2025 18:46 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "mongodb-ejjbon-backup" not found `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911184553 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911184553 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: datafile Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-vuvkk-mongodb-ejjbon-20250911184553-38be4 TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 18:45 UTC+0800 Completion Time: Sep 11,2025 18:46 UTC+0800 Status: Phase: Completed Total Size: 1984196 ActionSet Name: mongodb-physical-br Repository: backuprepo-kbcli-test Duration: 23s Start Time: Sep 11,2025 18:45 UTC+0800 Completion Time: Sep 11,2025 18:46 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/backup-ns-vuvkk-mongodb-ejjbon-20250911184553 Time Range Start: Sep 11,2025 18:46 UTC+0800 Time Range End: Sep 11,2025 18:46 UTC+0800 Warning Events: `kbcli cluster restore mongodb-ejjbon-backup --backup backup-ns-vuvkk-mongodb-ejjbon-20250911184553 --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup created check cluster status `kbcli cluster list mongodb-ejjbon-backup --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon-backup ns-vuvkk mongodb WipeOut Creating Sep 11,2025 18:46 UTC+0800 clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon-backup --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-backup-mongodb-0 ns-vuvkk mongodb-ejjbon-backup mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:46 UTC+0800 mongodb-ejjbon-backup-mongodb-1 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:47 UTC+0800 mongodb-ejjbon-backup-mongodb-2 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:48 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-backup-mongodb-0;secondary: mongodb-ejjbon-backup-mongodb-1 mongodb-ejjbon-backup-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911184553 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911184553 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: datafile Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-vuvkk-mongodb-ejjbon-20250911184553-38be4 TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 18:45 UTC+0800 Completion Time: Sep 11,2025 18:46 UTC+0800 Status: Phase: Completed Total Size: 1984196 ActionSet Name: mongodb-physical-br Repository: backuprepo-kbcli-test Duration: 23s Start Time: Sep 11,2025 18:45 UTC+0800 Completion Time: Sep 11,2025 18:46 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/backup-ns-vuvkk-mongodb-ejjbon-20250911184553 Time Range Start: Sep 11,2025 18:46 UTC+0800 Time Range End: Sep 11,2025 18:46 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo " echo \"rs.status()\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a952cea8ea2eaaa762f8 Connecting to: mongodb://@mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:47:58.731+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:48:03.853+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:48:03.853+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:48:03.853+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-backup-mongodb [direct: primary] admin> *** set: 'mongodb-ejjbon-backup-mongodb', date: ISODate('2025-09-11T10:49:59.232Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1757587798, i: 6 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-09-11T10:49:58.726Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1757587798, i: 6 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1757587798, i: 6 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1757587798, i: 6 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-09-11T10:49:58.726Z'), lastDurableWallTime: ISODate('2025-09-11T10:49:58.726Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1757587756, i: 1 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-09-11T10:49:27.859Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1757587756, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1757587756, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 2, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-09-11T10:49:27.874Z'), wMajorityWriteAvailabilityDate: ISODate('2025-09-11T10:49:28.383Z') ***, members: [ *** _id: 0, name: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 121, optime: *** ts: Timestamp(*** t: 1757587798, i: 6 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T10:49:58.000Z'), lastAppliedWallTime: ISODate('2025-09-11T10:49:58.726Z'), lastDurableWallTime: ISODate('2025-09-11T10:49:58.726Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: 'Could not find member to sync from', electionTime: Timestamp(*** t: 1757587767, i: 1 ***), electionDate: ISODate('2025-09-11T10:49:27.000Z'), configVersion: 1, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 42, optime: *** ts: Timestamp(*** t: 1757587793, i: 6 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757587793, i: 6 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T10:49:53.000Z'), optimeDurableDate: ISODate('2025-09-11T10:49:53.000Z'), lastAppliedWallTime: ISODate('2025-09-11T10:49:58.726Z'), lastDurableWallTime: ISODate('2025-09-11T10:49:58.726Z'), lastHeartbeat: ISODate('2025-09-11T10:49:58.127Z'), lastHeartbeatRecv: ISODate('2025-09-11T10:49:58.926Z'), pingMs: Long('2'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 ***, *** _id: 2, name: 'mongodb-ejjbon-backup-mongodb-2.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 42, optime: *** ts: Timestamp(*** t: 1757587793, i: 6 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757587793, i: 6 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T10:49:53.000Z'), optimeDurableDate: ISODate('2025-09-11T10:49:53.000Z'), lastAppliedWallTime: ISODate('2025-09-11T10:49:58.726Z'), lastDurableWallTime: ISODate('2025-09-11T10:49:58.726Z'), lastHeartbeat: ISODate('2025-09-11T10:49:58.127Z'), lastHeartbeatRecv: ISODate('2025-09-11T10:49:58.926Z'), pingMs: Long('2'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1757587798, i: 6 ***), signature: *** hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0), keyId: Long('0') *** ***, operationTime: Timestamp(*** t: 1757587798, i: 6 ***) *** mongodb-ejjbon-backup-mongodb [direct: primary] admin> connect cluster Success delete cluster mongodb-ejjbon-backup `kbcli cluster delete mongodb-ejjbon-backup --auto-approve --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup deleted pod_info:mongodb-ejjbon-backup-mongodb-0 4/4 Running 0 3m10s mongodb-ejjbon-backup-mongodb-1 4/4 Running 0 2m24s mongodb-ejjbon-backup-mongodb-2 4/4 Running 0 119s No resources found in ns-vuvkk namespace. delete cluster pod done No resources found in ns-vuvkk namespace. check cluster resource non-exist OK: pvc No resources found in ns-vuvkk namespace. delete cluster done No resources found in ns-vuvkk namespace. No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-ejjbon-rebuildinstance- namespace: ns-vuvkk spec: type: RebuildInstance clusterName: mongodb-ejjbon force: true rebuildFrom: - componentName: mongodb instances: - name: mongodb-ejjbon-mongodb-0 inPlace: true check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-ejjbon.yaml` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-rebuildinstance-b64rd created create test_ops_cluster_mongodb-ejjbon.yaml Success `rm -rf test_ops_cluster_mongodb-ejjbon.yaml` check ops status `kbcli cluster list-ops mongodb-ejjbon --status all --namespace ns-vuvkk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Running 0/1 Sep 11,2025 18:50 UTC+0800 check ops status done ops_status:mongodb-ejjbon-rebuildinstance-b64rd ns-vuvkk RebuildInstance mongodb-ejjbon mongodb Succeed 1/1 Sep 11,2025 18:50 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations mongodb-ejjbon-rebuildinstance-b64rd --namespace ns-vuvkk ` opsrequest.operations.kubeblocks.io/mongodb-ejjbon-rebuildinstance-b64rd patched `kbcli cluster delete-ops --name mongodb-ejjbon-rebuildinstance-b64rd --force --auto-approve --namespace ns-vuvkk ` OpsRequest mongodb-ejjbon-rebuildinstance-b64rd deleted check cluster status `kbcli cluster list mongodb-ejjbon --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon ns-vuvkk mongodb WipeOut Running Sep 11,2025 17:21 UTC+0800 app.kubernetes.io/instance=mongodb-ejjbon,clusterdefinition.kubeblocks.io/name=mongodb check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-mongodb-0 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:51 UTC+0800 mongodb-ejjbon-mongodb-1 ns-vuvkk mongodb-ejjbon mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:32 UTC+0800 mongodb-ejjbon-mongodb-2 ns-vuvkk mongodb-ejjbon mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:23 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-mongodb-2;secondary: mongodb-ejjbon-mongodb-0 mongodb-ejjbon-mongodb-1 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find()\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a9e853f036b04ca762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-mongodb-0 --namespace ns-vuvkk -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2a9f2ba4523af98a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb-ro.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T10:44:42.018+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:44:47.223+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:44:47.223+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:44:47.224+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: secondary] admin> [ *** _id: ObjectId('68c296429af82c94a0a762f9'), a: 'ozxfv' *** ] mongodb-ejjbon-mongodb [direct: secondary] admin> check cluster readonly data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check db_client batch data count `echo "echo \"db.executions_loop_table.estimatedDocumentCount();\" | mongosh --host mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin" | kubectl exec -it mongodb-ejjbon-mongodb-2 --namespace ns-vuvkk -- bash ` check db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-vuvkk-mongodb-ejjbon-20250911184553 --namespace ns-vuvkk ` backup.dataprotection.kubeblocks.io/backup-ns-vuvkk-mongodb-ejjbon-20250911184553 patched `kbcli cluster delete-backup mongodb-ejjbon --name backup-ns-vuvkk-mongodb-ejjbon-20250911184553 --force --auto-approve --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911184553 deleted No opsrequests found in ns-vuvkk namespace. `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `db.msg.drop();db.createCollection('msg');db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon0',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon1',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon2',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon3',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon4',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon5',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon6',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon7',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon8',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon9',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon10',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon11',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon12',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon13',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon14',time: new Date()***);db.msg.insertOne(***msg: 'kbcli-test-data-ejjbon15',time: new Date()***);` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2aa083c6c6129f1a762f8 Connecting to: mongodb://@mongodb-ejjbon-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:23:40.439+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:23:45.356+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:23:45.356+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:23:45.356+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-mongodb [direct: primary] admin> *** acknowledged: true, insertedId: ObjectId('68c2aa123c6c6129f1a76308') *** mongodb-ejjbon-mongodb [direct: primary] admin> cluster dump backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-fc6xr -n kb-heauh -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-fc6xr -n kb-heauh -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-heauh get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-heauh -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-heauh -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-heauh.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-heauh.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-8f45f86b6-jvj74 --namespace kb-heauh -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mongodb-ejjbon --method dump --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911185311 created successfully, you can view the progress: kbcli cluster list-backups --names=backup-ns-vuvkk-mongodb-ejjbon-20250911185311 -n ns-vuvkk check backup status `kbcli cluster list-backups mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-vuvkk-mongodb-ejjbon-20250911185311 ns-vuvkk mongodb-ejjbon dump Running Delete Sep 11,2025 18:53 UTC+0800 backup_status:mongodb-ejjbon-dump-Running backup_status:mongodb-ejjbon-dump-Running check backup status done backup_status:backup-ns-vuvkk-mongodb-ejjbon-20250911185311 ns-vuvkk mongodb-ejjbon dump Completed 46757 10s Delete Sep 11,2025 18:53 UTC+0800 Sep 11,2025 18:53 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "mongodb-ejjbon-backup" not found `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911185311 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911185311 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: dump Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-vuvkk-mongodb-ejjbon-20250911185311-73970 TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 18:53 UTC+0800 Completion Time: Sep 11,2025 18:53 UTC+0800 Status: Phase: Completed Total Size: 46757 ActionSet Name: mongodb-dump-br Repository: backuprepo-kbcli-test Duration: 10s Start Time: Sep 11,2025 18:53 UTC+0800 Completion Time: Sep 11,2025 18:53 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/backup-ns-vuvkk-mongodb-ejjbon-20250911185311 Time Range Start: Sep 11,2025 18:53 UTC+0800 Time Range End: Sep 11,2025 18:53 UTC+0800 Warning Events: `kbcli cluster restore mongodb-ejjbon-backup --backup backup-ns-vuvkk-mongodb-ejjbon-20250911185311 --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup created check cluster status `kbcli cluster list mongodb-ejjbon-backup --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon-backup ns-vuvkk mongodb WipeOut Creating Sep 11,2025 18:53 UTC+0800 clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon-backup --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-backup-mongodb-0 ns-vuvkk mongodb-ejjbon-backup mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:53 UTC+0800 mongodb-ejjbon-backup-mongodb-1 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 18:53 UTC+0800 mongodb-ejjbon-backup-mongodb-2 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 18:54 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-backup-mongodb-0;secondary: mongodb-ejjbon-backup-mongodb-1 mongodb-ejjbon-backup-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done check backup restore post ready post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: check backup restore post ready exists timeout check backup restore post ready done `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911185311 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911185311 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: dump Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-vuvkk-mongodb-ejjbon-20250911185311-73970 TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 18:53 UTC+0800 Completion Time: Sep 11,2025 18:53 UTC+0800 Status: Phase: Completed Total Size: 46757 ActionSet Name: mongodb-dump-br Repository: backuprepo-kbcli-test Duration: 10s Start Time: Sep 11,2025 18:53 UTC+0800 Completion Time: Sep 11,2025 18:53 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/backup-ns-vuvkk-mongodb-ejjbon-20250911185311 Time Range Start: Sep 11,2025 18:53 UTC+0800 Time Range End: Sep 11,2025 18:53 UTC+0800 Warning Events: `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `db.msg.find();` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2ab243185b9428ca762f8 Connecting to: mongodb://@mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:53:48.733+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:53:51.794+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:53:51.794+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:53:51.794+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-backup-mongodb [direct: primary] admin> [ *** _id: ObjectId('68c2aa103c6c6129f1a762f9'), msg: 'kbcli-test-data-ejjbon0', time: ISODate('2025-09-11T10:53:04.731Z') ***, *** _id: ObjectId('68c2aa103c6c6129f1a762fa'), msg: 'kbcli-test-data-ejjbon1', time: ISODate('2025-09-11T10:53:04.831Z') ***, *** _id: ObjectId('68c2aa103c6c6129f1a762fb'), msg: 'kbcli-test-data-ejjbon2', time: ISODate('2025-09-11T10:53:04.931Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a762fc'), msg: 'kbcli-test-data-ejjbon3', time: ISODate('2025-09-11T10:53:05.128Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a762fd'), msg: 'kbcli-test-data-ejjbon4', time: ISODate('2025-09-11T10:53:05.426Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a762fe'), msg: 'kbcli-test-data-ejjbon5', time: ISODate('2025-09-11T10:53:05.528Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a762ff'), msg: 'kbcli-test-data-ejjbon6', time: ISODate('2025-09-11T10:53:05.631Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a76300'), msg: 'kbcli-test-data-ejjbon7', time: ISODate('2025-09-11T10:53:05.727Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a76301'), msg: 'kbcli-test-data-ejjbon8', time: ISODate('2025-09-11T10:53:05.733Z') ***, *** _id: ObjectId('68c2aa113c6c6129f1a76302'), msg: 'kbcli-test-data-ejjbon9', time: ISODate('2025-09-11T10:53:05.927Z') ***, *** _id: ObjectId('68c2aa123c6c6129f1a76303'), msg: 'kbcli-test-data-ejjbon10', time: ISODate('2025-09-11T10:53:06.127Z') ***, *** _id: ObjectId('68c2aa123c6c6129f1a76304'), msg: 'kbcli-test-data-ejjbon11', time: ISODate('2025-09-11T10:53:06.227Z') ***, *** _id: ObjectId('68c2aa123c6c6129f1a76305'), msg: 'kbcli-test-data-ejjbon12', time: ISODate('2025-09-11T10:53:06.331Z') ***, *** _id: ObjectId('68c2aa123c6c6129f1a76306'), msg: 'kbcli-test-data-ejjbon13', time: ISODate('2025-09-11T10:53:06.427Z') ***, *** _id: ObjectId('68c2aa123c6c6129f1a76307'), msg: 'kbcli-test-data-ejjbon14', time: ISODate('2025-09-11T10:53:06.434Z') ***, *** _id: ObjectId('68c2aa123c6c6129f1a76308'), msg: 'kbcli-test-data-ejjbon15', time: ISODate('2025-09-11T10:53:06.627Z') *** ] mongodb-ejjbon-backup-mongodb [direct: primary] admin> dump backup check data Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo " echo \"rs.status()\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2ab2f09fc9331cba762f8 Connecting to: mongodb://@mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T10:53:48.733+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T10:53:51.794+00:00: You are running this process as the root user, which is not recommended 2025-09-11T10:53:51.794+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T10:53:51.794+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-backup-mongodb [direct: primary] admin> *** set: 'mongodb-ejjbon-backup-mongodb', date: ISODate('2025-09-11T10:57:57.326Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-09-11T10:57:55.231Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-09-11T10:57:55.231Z'), lastDurableWallTime: ISODate('2025-09-11T10:57:55.231Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1757588272, i: 2 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-09-11T10:53:53.226Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1757588032, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1757588032, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 1, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), newTermStartDate: ISODate('2025-09-11T10:53:53.358Z'), wMajorityWriteAvailabilityDate: ISODate('2025-09-11T10:53:53.413Z') ***, members: [ *** _id: 0, name: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 249, optime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T10:57:55.000Z'), lastAppliedWallTime: ISODate('2025-09-11T10:57:55.231Z'), lastDurableWallTime: ISODate('2025-09-11T10:57:55.231Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp(*** t: 1757588033, i: 1 ***), electionDate: ISODate('2025-09-11T10:53:53.000Z'), configVersion: 5, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 203, optime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T10:57:55.000Z'), optimeDurableDate: ISODate('2025-09-11T10:57:55.000Z'), lastAppliedWallTime: ISODate('2025-09-11T10:57:55.231Z'), lastDurableWallTime: ISODate('2025-09-11T10:57:55.231Z'), lastHeartbeat: ISODate('2025-09-11T10:57:57.027Z'), lastHeartbeatRecv: ISODate('2025-09-11T10:57:56.926Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 5, configTerm: 1 ***, *** _id: 2, name: 'mongodb-ejjbon-backup-mongodb-2.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 170, optime: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757588275, i: 2 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T10:57:55.000Z'), optimeDurableDate: ISODate('2025-09-11T10:57:55.000Z'), lastAppliedWallTime: ISODate('2025-09-11T10:57:55.231Z'), lastDurableWallTime: ISODate('2025-09-11T10:57:55.231Z'), lastHeartbeat: ISODate('2025-09-11T10:57:56.627Z'), lastHeartbeatRecv: ISODate('2025-09-11T10:57:56.728Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 1, infoMessage: '', configVersion: 5, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1757588275, i: 2 ***), signature: *** hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0), keyId: Long('0') *** ***, operationTime: Timestamp(*** t: 1757588275, i: 2 ***) *** mongodb-ejjbon-backup-mongodb [direct: primary] admin> connect cluster Success delete cluster mongodb-ejjbon-backup `kbcli cluster delete mongodb-ejjbon-backup --auto-approve --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup deleted pod_info:mongodb-ejjbon-backup-mongodb-0 4/4 Running 0 4m33s mongodb-ejjbon-backup-mongodb-1 4/4 Running 0 4m4s mongodb-ejjbon-backup-mongodb-2 4/4 Running 0 3m29s No resources found in ns-vuvkk namespace. delete cluster pod done No resources found in ns-vuvkk namespace. check cluster resource non-exist OK: pvc No resources found in ns-vuvkk namespace. delete cluster done No resources found in ns-vuvkk namespace. No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-vuvkk-mongodb-ejjbon-20250911185311 --namespace ns-vuvkk ` backup.dataprotection.kubeblocks.io/backup-ns-vuvkk-mongodb-ejjbon-20250911185311 patched `kbcli cluster delete-backup mongodb-ejjbon --name backup-ns-vuvkk-mongodb-ejjbon-20250911185311 --force --auto-approve --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911185311 deleted No opsrequests found in ns-vuvkk namespace. check cluster storage class cluster volume-snapshot backup `kbcli cluster backup mongodb-ejjbon --method volume-snapshot --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911185822 created successfully, you can view the progress: kbcli cluster list-backups --names=backup-ns-vuvkk-mongodb-ejjbon-20250911185822 -n ns-vuvkk check backup status `kbcli cluster list-backups mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-vuvkk-mongodb-ejjbon-20250911185822 ns-vuvkk mongodb-ejjbon volume-snapshot Running Delete Sep 11,2025 18:58 UTC+0800 backup_status:mongodb-ejjbon-volume-snapshot-Running backup_status:mongodb-ejjbon-volume-snapshot-Running backup_status:mongodb-ejjbon-volume-snapshot-Running check backup status done backup_status:backup-ns-vuvkk-mongodb-ejjbon-20250911185822 ns-vuvkk mongodb-ejjbon volume-snapshot Completed 4Gi 14s Delete Sep 11,2025 18:58 UTC+0800 Sep 11,2025 18:58 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "mongodb-ejjbon-backup" not found `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911185822 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911185822 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: volume-snapshot Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: createVolumeSnapshot-0: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2b40d9f] goroutine 1 [running]: github.com/apecloud/kbcli/pkg/cmd/dataprotection.PrintBackupObjDescribe(0xc0001c8a80, 0xc0009102c8) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:480 +0x4bf github.com/apecloud/kbcli/pkg/cmd/dataprotection.DescribeBackups(0xc0001c8a80, ***0xc001232cb0?, 0x192f1db?, 0xc000c85d48?***) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:458 +0x125 github.com/apecloud/kbcli/pkg/cmd/cluster.describeBackups(0x0?, ***0xc0008c5e40?, 0x0?, 0x8844954d00000000?***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:204 +0x66 github.com/apecloud/kbcli/pkg/cmd/cluster.NewDescribeBackupCmd.func1(0xc000fb8008?, ***0xc0008c5e40, 0x0, 0x4***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:195 +0xe5 github.com/spf13/cobra.(*Command).execute(0xc000fb8008, ***0xc0008c5e00, 0x4, 0x4***) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xa91 github.com/spf13/cobra.(*Command).ExecuteC(0xc000624308) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 k8s.io/component-base/cli.run(0xc000624308) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:146 +0x290 k8s.io/component-base/cli.RunNoErrOutput(...) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:84 main.main() /home/runner/work/kbcli/kbcli/cmd/cli/main.go:31 +0x18 `kbcli cluster restore mongodb-ejjbon-backup --backup backup-ns-vuvkk-mongodb-ejjbon-20250911185822 --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup created check cluster status `kbcli cluster list mongodb-ejjbon-backup --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon-backup ns-vuvkk mongodb WipeOut Creating Sep 11,2025 18:58 UTC+0800 clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon-backup --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-backup-mongodb-0 ns-vuvkk mongodb-ejjbon-backup mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 18:59 UTC+0800 mongodb-ejjbon-backup-mongodb-1 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 19:00 UTC+0800 mongodb-ejjbon-backup-mongodb-2 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 19:00 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-backup-mongodb-0;secondary: mongodb-ejjbon-backup-mongodb-1 mongodb-ejjbon-backup-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911185822 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911185822 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: volume-snapshot Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: createVolumeSnapshot-0: panic: runtime error: invalid memory address or nil pointer dereference [signal SIGSEGV: segmentation violation code=0x1 addr=0x28 pc=0x2b40d9f] goroutine 1 [running]: github.com/apecloud/kbcli/pkg/cmd/dataprotection.PrintBackupObjDescribe(0xc0010d6480, 0xc0008ac848) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:480 +0x4bf github.com/apecloud/kbcli/pkg/cmd/dataprotection.DescribeBackups(0xc0010d6480, ***0xc000f34fb0?, 0x192f1db?, 0xc000f2bd48?***) /home/runner/work/kbcli/kbcli/pkg/cmd/dataprotection/backup.go:458 +0x125 github.com/apecloud/kbcli/pkg/cmd/cluster.describeBackups(0x0?, ***0xc000521300?, 0x0?, 0x343983b700000000?***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:204 +0x66 github.com/apecloud/kbcli/pkg/cmd/cluster.NewDescribeBackupCmd.func1(0xc001239b08?, ***0xc000521300, 0x0, 0x4***) /home/runner/work/kbcli/kbcli/pkg/cmd/cluster/dataprotection.go:195 +0xe5 github.com/spf13/cobra.(*Command).execute(0xc001239b08, ***0xc000521200, 0x4, 0x4***) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:989 +0xa91 github.com/spf13/cobra.(*Command).ExecuteC(0xc000ba5208) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1117 +0x3ff github.com/spf13/cobra.(*Command).Execute(...) /home/runner/go/pkg/mod/github.com/spf13/cobra@v1.8.1/command.go:1041 k8s.io/component-base/cli.run(0xc000ba5208) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:146 +0x290 k8s.io/component-base/cli.RunNoErrOutput(...) /home/runner/go/pkg/mod/k8s.io/component-base@v0.29.2/cli/run.go:84 main.main() /home/runner/work/kbcli/kbcli/cmd/cli/main.go:31 +0x18 cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo " echo \"rs.status()\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2ac2950a2dd07e3a762f8 Connecting to: mongodb://@mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T11:00:28.340+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T11:00:33.762+00:00: You are running this process as the root user, which is not recommended 2025-09-11T11:00:33.762+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T11:00:33.762+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-backup-mongodb [direct: primary] admin> *** set: 'mongodb-ejjbon-backup-mongodb', date: ISODate('2025-09-11T11:02:06.733Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-09-11T11:02:02.027Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-09-11T11:02:02.027Z'), lastDurableWallTime: ISODate('2025-09-11T11:02:02.027Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1757588486, i: 1 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-09-11T11:01:36.405Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1757588486, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1757588486, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 2, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-09-11T11:01:36.436Z'), wMajorityWriteAvailabilityDate: ISODate('2025-09-11T11:01:36.934Z') ***, members: [ *** _id: 0, name: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 98, optime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T11:02:02.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:02:02.027Z'), lastDurableWallTime: ISODate('2025-09-11T11:02:02.027Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: 'Could not find member to sync from', electionTime: Timestamp(*** t: 1757588496, i: 1 ***), electionDate: ISODate('2025-09-11T11:01:36.000Z'), configVersion: 1, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 40, optime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T11:02:02.000Z'), optimeDurableDate: ISODate('2025-09-11T11:02:02.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:02:02.027Z'), lastDurableWallTime: ISODate('2025-09-11T11:02:02.027Z'), lastHeartbeat: ISODate('2025-09-11T11:02:06.428Z'), lastHeartbeatRecv: ISODate('2025-09-11T11:02:05.427Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 ***, *** _id: 2, name: 'mongodb-ejjbon-backup-mongodb-2.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 40, optime: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1757588522, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-09-11T11:02:02.000Z'), optimeDurableDate: ISODate('2025-09-11T11:02:02.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:02:02.027Z'), lastDurableWallTime: ISODate('2025-09-11T11:02:02.027Z'), lastHeartbeat: ISODate('2025-09-11T11:02:06.428Z'), lastHeartbeatRecv: ISODate('2025-09-11T11:02:05.427Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1757588522, i: 1 ***), signature: *** hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0), keyId: Long('0') *** ***, operationTime: Timestamp(*** t: 1757588522, i: 1 ***) *** mongodb-ejjbon-backup-mongodb [direct: primary] admin> connect cluster Success delete cluster mongodb-ejjbon-backup `kbcli cluster delete mongodb-ejjbon-backup --auto-approve --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup deleted pod_info:mongodb-ejjbon-backup-mongodb-0 4/4 Running 0 2m49s mongodb-ejjbon-backup-mongodb-1 4/4 Running 0 2m3s mongodb-ejjbon-backup-mongodb-2 4/4 Running 0 98s No resources found in ns-vuvkk namespace. delete cluster pod done No resources found in ns-vuvkk namespace. check cluster resource non-exist OK: pvc No resources found in ns-vuvkk namespace. delete cluster done No resources found in ns-vuvkk namespace. No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-vuvkk-mongodb-ejjbon-20250911185822 --namespace ns-vuvkk ` backup.dataprotection.kubeblocks.io/backup-ns-vuvkk-mongodb-ejjbon-20250911185822 patched `kbcli cluster delete-backup mongodb-ejjbon --name backup-ns-vuvkk-mongodb-ejjbon-20250911185822 --force --auto-approve --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911185822 deleted No opsrequests found in ns-vuvkk namespace. cluster pbm-physical backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-fc6xr -n kb-heauh -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-fc6xr -n kb-heauh -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-heauh get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-heauh -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-heauh -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-heauh.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-heauh.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-8f45f86b6-jvj74 --namespace kb-heauh -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mongodb-ejjbon --method pbm-physical --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911190235 created successfully, you can view the progress: kbcli cluster list-backups --names=backup-ns-vuvkk-mongodb-ejjbon-20250911190235 -n ns-vuvkk check backup status `kbcli cluster list-backups mongodb-ejjbon --namespace ns-vuvkk ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION DELETION-POLICY CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-vuvkk-mongodb-ejjbon-20250911190235 ns-vuvkk mongodb-ejjbon pbm-physical Running Delete Sep 11,2025 19:02 UTC+0800 backup_status:mongodb-ejjbon-pbm-physical-Running backup_status:mongodb-ejjbon-pbm-physical-Running backup_status:mongodb-ejjbon-pbm-physical-Running backup_status:mongodb-ejjbon-pbm-physical-Running backup_status:mongodb-ejjbon-pbm-physical-Running backup_status:mongodb-ejjbon-pbm-physical-Running check backup status done backup_status:backup-ns-vuvkk-mongodb-ejjbon-20250911190235 ns-vuvkk mongodb-ejjbon pbm-physical Completed 593101 31s Delete Sep 11,2025 19:02 UTC+0800 Sep 11,2025 19:03 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "mongodb-ejjbon-backup" not found `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911190235 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911190235 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: pbm-physical Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-vuvkk-mongodb-ejjbon-20250911190235-6cc0b TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 19:02 UTC+0800 Completion Time: Sep 11,2025 19:03 UTC+0800 Extras: =================== 1 =================== backupName: 2025-09-11T11:02:54Z backupType: physical lastWriteTime: 2025-09-11T11:02:56Z Status: Phase: Completed Total Size: 593101 ActionSet Name: mongodb-rs-pbm-physical Repository: backuprepo-kbcli-test Duration: 31s Start Time: Sep 11,2025 19:02 UTC+0800 Completion Time: Sep 11,2025 19:03 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/backup-ns-vuvkk-mongodb-ejjbon-20250911190235 Time Range Start: Sep 11,2025 19:02 UTC+0800 Time Range End: Sep 11,2025 19:02 UTC+0800 Warning Events: `kbcli cluster restore mongodb-ejjbon-backup --backup backup-ns-vuvkk-mongodb-ejjbon-20250911190235 --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup created check cluster status `kbcli cluster list mongodb-ejjbon-backup --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon-backup ns-vuvkk mongodb WipeOut Creating Sep 11,2025 19:03 UTC+0800 clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon-backup --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-backup-mongodb-0 ns-vuvkk mongodb-ejjbon-backup mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 19:03 UTC+0800 mongodb-ejjbon-backup-mongodb-1 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 19:04 UTC+0800 mongodb-ejjbon-backup-mongodb-2 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 19:04 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-backup-mongodb-0;secondary: mongodb-ejjbon-backup-mongodb-1 mongodb-ejjbon-backup-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done check backup restore post ready post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: post_ready_pod_exists: check backup restore post ready exists timeout check backup restore post ready done `kbcli cluster describe-backup --names backup-ns-vuvkk-mongodb-ejjbon-20250911190235 --namespace ns-vuvkk ` Name: backup-ns-vuvkk-mongodb-ejjbon-20250911190235 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: pbm-physical Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-backup-ns-vuvkk-mongodb-ejjbon-20250911190235-6cc0b TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 19:02 UTC+0800 Completion Time: Sep 11,2025 19:03 UTC+0800 Extras: =================== 1 =================== lastWriteTime: 2025-09-11T11:02:56Z backupName: 2025-09-11T11:02:54Z backupType: physical Status: Phase: Completed Total Size: 593101 ActionSet Name: mongodb-rs-pbm-physical Repository: backuprepo-kbcli-test Duration: 31s Start Time: Sep 11,2025 19:02 UTC+0800 Completion Time: Sep 11,2025 19:03 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/backup-ns-vuvkk-mongodb-ejjbon-20250911190235 Time Range Start: Sep 11,2025 19:02 UTC+0800 Time Range End: Sep 11,2025 19:02 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo " echo \"rs.status()\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file MongoNetworkError: connect ECONNREFUSED 10.0.242.248:27017 command terminated with exit code 1 check connect cluster... Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file MongoNetworkError: connect ECONNREFUSED 10.0.242.248:27017 command terminated with exit code 1 check connect cluster... Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file MongoNetworkError: connect ECONNREFUSED 10.0.242.248:27017 command terminated with exit code 1 check connect cluster... Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2ada2f7849e3b38a762f8 Connecting to: mongodb://@mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 mongosh 2.5.8 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-09-11T11:07:57.233+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T11:08:02.785+00:00: You are running this process as the root user, which is not recommended 2025-09-11T11:08:02.785+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T11:08:02.785+00:00: vm.max_map_count is too low ------ mongodb-ejjbon-backup-mongodb [direct: primary] admin> *** set: 'mongodb-ejjbon-backup-mongodb', date: ISODate('2025-09-11T11:08:23.122Z'), myState: 1, term: Long('18'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, lastCommittedWallTime: ISODate('2025-09-11T11:08:19.739Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, appliedOpTime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, durableOpTime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, lastAppliedWallTime: ISODate('2025-09-11T11:08:19.739Z'), lastDurableWallTime: ISODate('2025-09-11T11:08:19.739Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1757588893, i: 2 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-09-11T11:08:13.660Z'), electionTerm: Long('18'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 0, i: 0 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1757588576, i: 1 ***), t: Long('17') ***, numVotesNeeded: 2, priorityAtElection: 1, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-09-11T11:08:13.687Z'), wMajorityWriteAvailabilityDate: ISODate('2025-09-11T11:08:14.186Z') ***, members: [ *** _id: 0, name: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 17, optime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, optimeDurable: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, optimeDate: ISODate('2025-09-11T11:08:19.000Z'), optimeDurableDate: ISODate('2025-09-11T11:08:19.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:08:19.739Z'), lastDurableWallTime: ISODate('2025-09-11T11:08:19.739Z'), lastHeartbeat: ISODate('2025-09-11T11:08:21.717Z'), lastHeartbeatRecv: ISODate('2025-09-11T11:08:22.726Z'), pingMs: Long('9'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-2.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 2, infoMessage: '', configVersion: 1, configTerm: -1 ***, *** _id: 1, name: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 19, optime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, optimeDurable: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, optimeDate: ISODate('2025-09-11T11:08:19.000Z'), optimeDurableDate: ISODate('2025-09-11T11:08:19.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:08:19.739Z'), lastDurableWallTime: ISODate('2025-09-11T11:08:19.739Z'), lastHeartbeat: ISODate('2025-09-11T11:08:21.688Z'), lastHeartbeatRecv: ISODate('2025-09-11T11:08:22.187Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: -1 ***, *** _id: 2, name: 'mongodb-ejjbon-backup-mongodb-2.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 26, optime: *** ts: Timestamp(*** t: 1757588899, i: 3 ***), t: Long('18') ***, optimeDate: ISODate('2025-09-11T11:08:19.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:08:19.739Z'), lastDurableWallTime: ISODate('2025-09-11T11:08:19.739Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: 'Could not find member to sync from', electionTime: Timestamp(*** t: 1757588893, i: 1 ***), electionDate: ISODate('2025-09-11T11:08:13.000Z'), configVersion: 1, configTerm: -1, self: true, lastHeartbeatMessage: '' *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1757588899, i: 3 ***), signature: *** hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0), keyId: Long('0') *** ***, operationTime: Timestamp(*** t: 1757588899, i: 3 ***) *** mongodb-ejjbon-backup-mongodb [direct: primary] admin> connect cluster Success delete cluster mongodb-ejjbon-backup `kbcli cluster delete mongodb-ejjbon-backup --auto-approve --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup deleted pod_info:mongodb-ejjbon-backup-mongodb-0 4/4 Running 0 4m32s mongodb-ejjbon-backup-mongodb-1 4/4 Running 0 4m18s mongodb-ejjbon-backup-mongodb-2 4/4 Running 0 3m53s No resources found in ns-vuvkk namespace. delete cluster pod done No resources found in ns-vuvkk namespace. check cluster resource non-exist OK: pvc No resources found in ns-vuvkk namespace. delete cluster done No resources found in ns-vuvkk namespace. No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-vuvkk-mongodb-ejjbon-20250911190235 --namespace ns-vuvkk ` backup.dataprotection.kubeblocks.io/backup-ns-vuvkk-mongodb-ejjbon-20250911190235 patched `kbcli cluster delete-backup mongodb-ejjbon --name backup-ns-vuvkk-mongodb-ejjbon-20250911190235 --force --auto-approve --namespace ns-vuvkk ` Backup backup-ns-vuvkk-mongodb-ejjbon-20250911190235 deleted No opsrequests found in ns-vuvkk namespace. `kubectl get backupschedule -l app.kubernetes.io/instance=mongodb-ejjbon ` `kubectl get backupschedule mongodb-ejjbon-mongodb-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched check backup status `kbcli cluster list-backups mongodb-ejjbon --namespace ns-vuvkk ` No backups found in ns-vuvkk namespace. No backups found in ns-vuvkk namespace. backup_status:-- No backups found in ns-vuvkk namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=mongodb-ejjbon ` `kubectl get backupschedule mongodb-ejjbon-mongodb-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backup_status:mongodb-ejjbon-pbm-physical-Running backup_status:mongodb-ejjbon-pbm-physical-Running check backup status done backup_status:mongodb-ejjbon-pbm-physical-20250911110900 ns-vuvkk mongodb-ejjbon pbm-physical Completed 551108 15s Delete Sep 11,2025 19:09 UTC+0800 Sep 11,2025 19:09 UTC+0800 Sep 18,2025 19:09 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=mongodb-ejjbon ` `kubectl get backupschedule mongodb-ejjbon-mongodb-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-ejjbon-mongodb-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.operations.kubeblocks.io "mongodb-ejjbon-backup" not found `kbcli cluster describe-backup --names mongodb-ejjbon-pbm-physical-20250911110900 --namespace ns-vuvkk ` Name: mongodb-ejjbon-pbm-physical-20250911110900 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: pbm-physical Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-mongodb-ejjbon-pbm-physical-20250911110900-d90e0c41 TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 19:09 UTC+0800 Completion Time: Sep 11,2025 19:09 UTC+0800 Extras: =================== 1 =================== backupType: physical lastWriteTime: 2025-09-11T11:09:05Z backupName: 2025-09-11T11:09:03Z Status: Phase: Completed Total Size: 551108 ActionSet Name: mongodb-rs-pbm-physical Repository: backuprepo-kbcli-test Duration: 15s Expiration Time: Sep 18,2025 19:09 UTC+0800 Start Time: Sep 11,2025 19:09 UTC+0800 Completion Time: Sep 11,2025 19:09 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/mongodb-ejjbon-pbm-physical-20250911110900 Time Range Start: Sep 11,2025 19:09 UTC+0800 Time Range End: Sep 11,2025 19:09 UTC+0800 Warning Events: `kbcli cluster restore mongodb-ejjbon-backup --backup mongodb-ejjbon-pbm-physical-20250911110900 --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup created check cluster status `kbcli cluster list mongodb-ejjbon-backup --show-labels --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-ejjbon-backup ns-vuvkk mongodb WipeOut Creating Sep 11,2025 19:09 UTC+0800 clusterdefinition.kubeblocks.io/name=mongodb cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-ejjbon-backup --namespace ns-vuvkk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-ejjbon-backup-mongodb-0 ns-vuvkk mongodb-ejjbon-backup mongodb Running primary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000001/10.224.0.7 Sep 11,2025 19:09 UTC+0800 mongodb-ejjbon-backup-mongodb-1 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000000/10.224.0.6 Sep 11,2025 19:10 UTC+0800 mongodb-ejjbon-backup-mongodb-2 ns-vuvkk mongodb-ejjbon-backup mongodb Running secondary 0 200m / 200m 644245094400m / 644245094400m data:4Gi aks-cicdamdpool-40497330-vmss000002/10.224.0.5 Sep 11,2025 19:10 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-ejjbon-backup-mongodb-0;secondary: mongodb-ejjbon-backup-mongodb-1 mongodb-ejjbon-backup-mongodb-2 `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. check cluster connect `echo " echo \"\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash` check cluster connect done check backup restore post ready check backup restore post ready exists post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 23s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 11s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 27s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 15s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 33s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 21s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 38s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 26s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 43s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 31s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 49s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 37s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 54s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 42s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 59s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 47s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 65s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 53s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 70s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 58s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 75s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 63s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 81s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 69s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 86s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 74s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 91s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 79s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 97s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 85s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 102s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 90s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 107s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 95s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 113s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 101s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 118s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 106s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m3s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 111s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m9s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 117s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m14s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m2s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m20s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m8s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m25s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m13s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m30s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m18s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m36s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m24s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m41s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m29s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m46s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m34s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m52s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m40s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 2m57s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m45s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m2s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m50s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m8s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 2m56s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m13s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m1s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m18s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m6s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m24s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m12s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m29s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m17s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m34s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m22s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m40s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m28s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m45s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m33s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m51s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m39s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 3m56s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m44s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m1s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m49s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m7s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 3m55s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m12s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m17s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m5s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m23s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m11s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m28s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m16s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m33s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m21s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m39s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m27s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m44s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m32s post_ready_pod_status:restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-6ck6h 0/1 Error 0 4m50s restore-post-ready-32764a8b-mongodb-ejjbon-pbm-physical-2-j8knc 1/1 Running 0 4m38s check backup restore post ready done `kbcli cluster describe-backup --names mongodb-ejjbon-pbm-physical-20250911110900 --namespace ns-vuvkk ` Name: mongodb-ejjbon-pbm-physical-20250911110900 Cluster: mongodb-ejjbon Namespace: ns-vuvkk Spec: Method: pbm-physical Policy Name: mongodb-ejjbon-mongodb-backup-policy Actions: dp-backup-0: ActionType: Job WorkloadName: dp-backup-0-mongodb-ejjbon-pbm-physical-20250911110900-d90e0c41 TargetPodName: mongodb-ejjbon-mongodb-0 Phase: Completed Start Time: Sep 11,2025 19:09 UTC+0800 Completion Time: Sep 11,2025 19:09 UTC+0800 Extras: =================== 1 =================== lastWriteTime: 2025-09-11T11:09:05Z backupName: 2025-09-11T11:09:03Z backupType: physical Status: Phase: Completed Total Size: 551108 ActionSet Name: mongodb-rs-pbm-physical Repository: backuprepo-kbcli-test Duration: 15s Expiration Time: Sep 18,2025 19:09 UTC+0800 Start Time: Sep 11,2025 19:09 UTC+0800 Completion Time: Sep 11,2025 19:09 UTC+0800 Path: /ns-vuvkk/mongodb-ejjbon-41c6c781-4d35-4c9c-9b22-0086ea1aefae/mongodb/mongodb-ejjbon-pbm-physical-20250911110900 Time Range Start: Sep 11,2025 19:09 UTC+0800 Time Range End: Sep 11,2025 19:09 UTC+0800 Warning Events: cluster connect `kubectl get secrets -l app.kubernetes.io/instance=mongodb-ejjbon-backup` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-ejjbon-backup-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:16yFD9v3AK34W9u1;DB_PORT:27017;DB_DATABASE: No resources found in ns-vuvkk namespace. `echo " echo \"rs.status()\" | mongosh --host mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local --port 27017 -u root -p 16yFD9v3AK34W9u1 --authenticationDatabase admin admin " | kubectl exec -it mongodb-ejjbon-backup-mongodb-0 --namespace ns-vuvkk -- bash ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file MongoNetworkError: connect ECONNREFUSED 10.0.130.197:27017 command terminated with exit code 1 check connect cluster... Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68c2af99d3d7266d8aa762f8 Connecting to: mongodb://@mongodb-ejjbon-backup-mongodb-mongodb.ns-vuvkk.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.5.0 Using MongoDB: 7.0.18-11 Using Mongosh: 2.5.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-09-11T11:16:18.301+00:00: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine. See http://dochub.mongodb.org/core/prodnotes-filesystem 2025-09-11T11:16:23.600+00:00: You are running this process as the root user, which is not recommended 2025-09-11T11:16:23.600+00:00: For customers running MongoDB 7.0, we suggest changing the contents of the following sysfsFile 2025-09-11T11:16:23.600+00:00: vm.max_map_count is too low ------ admin> *** set: 'mongodb-ejjbon-backup-mongodb', date: ISODate('2025-09-11T11:16:49.122Z'), myState: 1, term: Long('18'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, lastCommittedWallTime: ISODate('2025-09-11T11:16:45.496Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, appliedOpTime: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, durableOpTime: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, lastAppliedWallTime: ISODate('2025-09-11T11:16:45.496Z'), lastDurableWallTime: ISODate('2025-09-11T11:16:45.496Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1757589394, i: 2 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-09-11T11:16:34.724Z'), electionTerm: Long('18'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 0, i: 0 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1757588945, i: 1 ***), t: Long('17') ***, numVotesNeeded: 2, priorityAtElection: 1, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-09-11T11:16:34.752Z'), wMajorityWriteAvailabilityDate: ISODate('2025-09-11T11:16:35.249Z') ***, members: [ *** _id: 0, name: 'mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 0, state: 8, stateStr: '(not reachable/healthy)', uptime: 0, optime: *** ts: Timestamp(*** t: 0, i: 0 ***), t: Long('-1') ***, optimeDurable: *** ts: Timestamp(*** t: 0, i: 0 ***), t: Long('-1') ***, optimeDate: ISODate('1970-01-01T00:00:00.000Z'), optimeDurableDate: ISODate('1970-01-01T00:00:00.000Z'), lastAppliedWallTime: ISODate('1970-01-01T00:00:00.000Z'), lastDurableWallTime: ISODate('1970-01-01T00:00:00.000Z'), lastHeartbeat: ISODate('2025-09-11T11:16:47.335Z'), lastHeartbeatRecv: ISODate('1970-01-01T00:00:00.000Z'), pingMs: Long('0'), lastHeartbeatMessage: 'Error connecting to mongodb-ejjbon-backup-mongodb-0.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017 (10.244.3.155:27017) :: caused by :: onInvoke :: caused by :: Connection refused', syncSourceHost: '', syncSourceId: -1, infoMessage: '', configVersion: -1, configTerm: -1 ***, *** _id: 1, name: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 31, optime: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, optimeDate: ISODate('2025-09-11T11:16:45.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:16:45.496Z'), lastDurableWallTime: ISODate('2025-09-11T11:16:45.496Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: 'Could not find member to sync from', electionTime: Timestamp(*** t: 1757589394, i: 1 ***), electionDate: ISODate('2025-09-11T11:16:34.000Z'), configVersion: 2, configTerm: 18, self: true, lastHeartbeatMessage: '' ***, *** _id: 2, name: 'mongodb-ejjbon-backup-mongodb-2.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 23, optime: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, optimeDurable: *** ts: Timestamp(*** t: 1757589405, i: 2 ***), t: Long('18') ***, optimeDate: ISODate('2025-09-11T11:16:45.000Z'), optimeDurableDate: ISODate('2025-09-11T11:16:45.000Z'), lastAppliedWallTime: ISODate('2025-09-11T11:16:45.496Z'), lastDurableWallTime: ISODate('2025-09-11T11:16:45.496Z'), lastHeartbeat: ISODate('2025-09-11T11:16:47.221Z'), lastHeartbeatRecv: ISODate('2025-09-11T11:16:47.221Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-ejjbon-backup-mongodb-1.mongodb-ejjbon-backup-mongodb-headless.ns-vuvkk.svc:27017', syncSourceId: 1, infoMessage: '', configVersion: 2, configTerm: 18 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1757589405, i: 2 ***), signature: *** hash: Binary.createFromBase64('AAAAAAAAAAAAAAAAAAAAAAAAAAA=', 0), keyId: Long('0') *** ***, operationTime: Timestamp(*** t: 1757589405, i: 2 ***) *** mongodb-ejjbon-backup-mongodb [direct: primary] admin> connect cluster Success delete cluster mongodb-ejjbon-backup `kbcli cluster delete mongodb-ejjbon-backup --auto-approve --namespace ns-vuvkk ` Cluster mongodb-ejjbon-backup deleted pod_info:mongodb-ejjbon-backup-mongodb-0 4/4 Running 1 (3m26s ago) 7m6s mongodb-ejjbon-backup-mongodb-1 4/4 Running 0 6m25s mongodb-ejjbon-backup-mongodb-2 4/4 Running 0 6m5s No resources found in ns-vuvkk namespace. delete cluster pod done No resources found in ns-vuvkk namespace. check cluster resource non-exist OK: pvc No resources found in ns-vuvkk namespace. delete cluster done No resources found in ns-vuvkk namespace. No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups mongodb-ejjbon-pbm-physical-20250911110900 --namespace ns-vuvkk ` backup.dataprotection.kubeblocks.io/mongodb-ejjbon-pbm-physical-20250911110900 patched `kbcli cluster delete-backup mongodb-ejjbon --name mongodb-ejjbon-pbm-physical-20250911110900 --force --auto-approve --namespace ns-vuvkk ` Backup mongodb-ejjbon-pbm-physical-20250911110900 deleted No opsrequests found in ns-vuvkk namespace. cluster list-logs `kbcli cluster list-logs mongodb-ejjbon --namespace ns-vuvkk ` No log files found. Error from server (NotFound): pods "mongodb-ejjbon-mongodb-2" not found cluster logs `kbcli cluster logs mongodb-ejjbon --tail 30 --namespace ns-vuvkk ` Defaulted container "mongodb" out of: mongodb, mongodb-backup-agent, exporter, kbagent, init-syncer (init), init-kubectl (init), init-pbm-agent (init), kbagent-worker (init) 2025-09-11T11:02:57Z INFO HA This member is Cluster's leader 2025-09-11T11:02:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:03:57Z INFO HA This member is Cluster's leader 2025-09-11T11:03:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:04:57Z INFO HA This member is Cluster's leader 2025-09-11T11:04:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:05:57Z INFO HA This member is Cluster's leader 2025-09-11T11:05:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:06:57Z INFO HA This member is Cluster's leader 2025-09-11T11:06:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:07:57Z INFO HA This member is Cluster's leader 2025-09-11T11:07:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:08:57Z INFO HA This member is Cluster's leader 2025-09-11T11:08:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:09:57Z INFO HA This member is Cluster's leader 2025-09-11T11:09:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:10:57Z INFO HA This member is Cluster's leader 2025-09-11T11:10:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:11:57Z INFO HA This member is Cluster's leader 2025-09-11T11:11:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:12:57Z INFO HA This member is Cluster's leader 2025-09-11T11:12:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:13:57Z INFO HA This member is Cluster's leader 2025-09-11T11:13:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:14:57Z INFO HA This member is Cluster's leader 2025-09-11T11:14:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:15:57Z INFO HA This member is Cluster's leader 2025-09-11T11:15:57Z DEBUG HA Refresh leader ttl 2025-09-11T11:16:57Z INFO HA This member is Cluster's leader 2025-09-11T11:16:57Z DEBUG HA Refresh leader ttl cluster logs running `kbcli cluster logs mongodb-ejjbon --tail 30 --file-type=running --namespace ns-vuvkk ` ==> /data/mongodb/logs/mongodb.log <== ***"t":***"$date":"2025-09-11T11:17:16.751+00:00"***,"s":"I", "c":"-", "id":20883, "ctx":"conn45337","msg":"Interrupted operation as its client disconnected","attr":***"opId":224892*** ***"t":***"$date":"2025-09-11T11:17:16.751+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45336","msg":"Connection ended","attr":***"remote":"10.244.4.15:33404","uuid":***"uuid":***"$uuid":"6e25b71e-e9da-4232-8a37-30dc62f2edd3"***,"connectionId":45336,"connectionCount":42*** ***"t":***"$date":"2025-09-11T11:17:16.751+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45338","msg":"Connection ended","attr":***"remote":"10.244.4.15:33428","uuid":***"uuid":***"$uuid":"f47b21f1-3131-4e32-9568-e1225411de16"***,"connectionId":45338,"connectionCount":41*** ***"t":***"$date":"2025-09-11T11:17:16.752+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45337","msg":"Connection ended","attr":***"remote":"10.244.4.15:33420","uuid":***"uuid":***"$uuid":"35ec755b-2419-4d46-9658-4cbec0198fa6"***,"connectionId":45337,"connectionCount":40*** ***"t":***"$date":"2025-09-11T11:17:17.152+00:00"***,"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":***"remote":"10.244.3.191:56066","uuid":***"uuid":***"$uuid":"2a328cc3-334a-4a4f-b76a-695514991c94"***,"connectionId":45339,"connectionCount":41*** ***"t":***"$date":"2025-09-11T11:17:17.153+00:00"***,"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":***"remote":"10.244.3.191:56070","uuid":***"uuid":***"$uuid":"b421c30b-11e7-4548-bd90-d55ab735ff61"***,"connectionId":45340,"connectionCount":42*** ***"t":***"$date":"2025-09-11T11:17:17.153+00:00"***,"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn45339","msg":"client metadata","attr":***"remote":"10.244.3.191:56066","client":"conn45339","negotiatedCompressors":[],"doc":***"driver":***"name":"mongo-go-driver","version":"v1.11.6"***,"os":***"type":"linux","architecture":"amd64"***,"platform":"go1.23.10"*** ***"t":***"$date":"2025-09-11T11:17:17.153+00:00"***,"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn45340","msg":"client metadata","attr":***"remote":"10.244.3.191:56070","client":"conn45340","negotiatedCompressors":[],"doc":***"driver":***"name":"mongo-go-driver","version":"v1.11.6"***,"os":***"type":"linux","architecture":"amd64"***,"platform":"go1.23.10"*** ***"t":***"$date":"2025-09-11T11:17:17.158+00:00"***,"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":***"remote":"10.244.3.191:56082","uuid":***"uuid":***"$uuid":"98f151ee-4737-40a1-98d2-bc1c6baacefe"***,"connectionId":45341,"connectionCount":43*** ***"t":***"$date":"2025-09-11T11:17:17.158+00:00"***,"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn45341","msg":"client metadata","attr":***"remote":"10.244.3.191:56082","client":"conn45341","negotiatedCompressors":[],"doc":***"driver":***"name":"mongo-go-driver","version":"v1.11.6"***,"os":***"type":"linux","architecture":"amd64"***,"platform":"go1.23.10"*** ***"t":***"$date":"2025-09-11T11:17:17.158+00:00"***,"s":"I", "c":"ACCESS", "id":6788604, "ctx":"conn45341","msg":"Auth metrics report","attr":***"metric":"acquireUser","micros":0*** ***"t":***"$date":"2025-09-11T11:17:17.168+00:00"***,"s":"I", "c":"ACCESS", "id":5286306, "ctx":"conn45341","msg":"Successfully authenticated","attr":***"client":"10.244.3.191:56082","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"root","db":"admin","result":0,"metrics":***"conversation_duration":***"micros":9580,"summary":***"0":***"step":1,"step_total":2,"duration_micros":46***,"1":***"step":2,"step_total":2,"duration_micros":29***,"extraInfo":*** ***"t":***"$date":"2025-09-11T11:17:17.168+00:00"***,"s":"I", "c":"NETWORK", "id":6788700, "ctx":"conn45341","msg":"Received first command on ingress connection since session start or auth handshake","attr":***"elapsedMillis":0*** ***"t":***"$date":"2025-09-11T11:17:17.169+00:00"***,"s":"I", "c":"-", "id":20883, "ctx":"conn45340","msg":"Interrupted operation as its client disconnected","attr":***"opId":224912*** ***"t":***"$date":"2025-09-11T11:17:17.169+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45341","msg":"Connection ended","attr":***"remote":"10.244.3.191:56082","uuid":***"uuid":***"$uuid":"98f151ee-4737-40a1-98d2-bc1c6baacefe"***,"connectionId":45341,"connectionCount":42*** ***"t":***"$date":"2025-09-11T11:17:17.170+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45339","msg":"Connection ended","attr":***"remote":"10.244.3.191:56066","uuid":***"uuid":***"$uuid":"2a328cc3-334a-4a4f-b76a-695514991c94"***,"connectionId":45339,"connectionCount":41*** ***"t":***"$date":"2025-09-11T11:17:17.170+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45340","msg":"Connection ended","attr":***"remote":"10.244.3.191:56070","uuid":***"uuid":***"$uuid":"b421c30b-11e7-4548-bd90-d55ab735ff61"***,"connectionId":45340,"connectionCount":40*** ***"t":***"$date":"2025-09-11T11:17:17.222+00:00"***,"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":***"remote":"10.244.3.191:56104","uuid":***"uuid":***"$uuid":"97e5b472-1314-481e-9e2d-d486a2cb8ac4"***,"connectionId":45342,"connectionCount":41*** ***"t":***"$date":"2025-09-11T11:17:17.222+00:00"***,"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":***"remote":"10.244.3.191:56092","uuid":***"uuid":***"$uuid":"e7920f51-5ae2-4a84-b4ae-1a6ca031cb10"***,"connectionId":45343,"connectionCount":42*** ***"t":***"$date":"2025-09-11T11:17:17.222+00:00"***,"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn45342","msg":"client metadata","attr":***"remote":"10.244.3.191:56104","client":"conn45342","negotiatedCompressors":[],"doc":***"driver":***"name":"mongo-go-driver","version":"v1.11.6"***,"os":***"type":"linux","architecture":"amd64"***,"platform":"go1.23.10"*** ***"t":***"$date":"2025-09-11T11:17:17.222+00:00"***,"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn45343","msg":"client metadata","attr":***"remote":"10.244.3.191:56092","client":"conn45343","negotiatedCompressors":[],"doc":***"driver":***"name":"mongo-go-driver","version":"v1.11.6"***,"os":***"type":"linux","architecture":"amd64"***,"platform":"go1.23.10"*** ***"t":***"$date":"2025-09-11T11:17:17.227+00:00"***,"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":***"remote":"10.244.3.191:56112","uuid":***"uuid":***"$uuid":"723e9436-923a-42d5-8d4d-53596327f967"***,"connectionId":45344,"connectionCount":43*** ***"t":***"$date":"2025-09-11T11:17:17.228+00:00"***,"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn45344","msg":"client metadata","attr":***"remote":"10.244.3.191:56112","client":"conn45344","negotiatedCompressors":[],"doc":***"driver":***"name":"mongo-go-driver","version":"v1.11.6"***,"os":***"type":"linux","architecture":"amd64"***,"platform":"go1.23.10"*** ***"t":***"$date":"2025-09-11T11:17:17.228+00:00"***,"s":"I", "c":"ACCESS", "id":6788604, "ctx":"conn45344","msg":"Auth metrics report","attr":***"metric":"acquireUser","micros":0*** ***"t":***"$date":"2025-09-11T11:17:17.238+00:00"***,"s":"I", "c":"ACCESS", "id":5286306, "ctx":"conn45344","msg":"Successfully authenticated","attr":***"client":"10.244.3.191:56112","isSpeculative":true,"isClusterMember":false,"mechanism":"SCRAM-SHA-256","user":"root","db":"admin","result":0,"metrics":***"conversation_duration":***"micros":10757,"summary":***"0":***"step":1,"step_total":2,"duration_micros":48***,"1":***"step":2,"step_total":2,"duration_micros":31***,"extraInfo":*** ***"t":***"$date":"2025-09-11T11:17:17.239+00:00"***,"s":"I", "c":"NETWORK", "id":6788700, "ctx":"conn45344","msg":"Received first command on ingress connection since session start or auth handshake","attr":***"elapsedMillis":0*** ***"t":***"$date":"2025-09-11T11:17:17.240+00:00"***,"s":"I", "c":"-", "id":20883, "ctx":"conn45342","msg":"Interrupted operation as its client disconnected","attr":***"opId":224920*** ***"t":***"$date":"2025-09-11T11:17:17.240+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45344","msg":"Connection ended","attr":***"remote":"10.244.3.191:56112","uuid":***"uuid":***"$uuid":"723e9436-923a-42d5-8d4d-53596327f967"***,"connectionId":45344,"connectionCount":42*** ***"t":***"$date":"2025-09-11T11:17:17.240+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45343","msg":"Connection ended","attr":***"remote":"10.244.3.191:56092","uuid":***"uuid":***"$uuid":"e7920f51-5ae2-4a84-b4ae-1a6ca031cb10"***,"connectionId":45343,"connectionCount":41*** ***"t":***"$date":"2025-09-11T11:17:17.240+00:00"***,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn45342","msg":"Connection ended","attr":***"remote":"10.244.3.191:56104","uuid":***"uuid":***"$uuid":"97e5b472-1314-481e-9e2d-d486a2cb8ac4"***,"connectionId":45342,"connectionCount":40*** ==> /data/mongodb/logs/mongodb.log.2025-09-11T09-29-40 <== ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"CONTROL", "id":4784928, "ctx":"SignalHandler","msg":"Shutting down the TTL monitor"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"INDEX", "id":3684100, "ctx":"SignalHandler","msg":"Shutting down TTL collection monitor thread"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"INDEX", "id":3684101, "ctx":"SignalHandler","msg":"Finished shutting down TTL collection monitor thread"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"CONTROL", "id":6278511, "ctx":"SignalHandler","msg":"Shutting down the Change Stream Expired Pre-images Remover"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"QUERY", "id":6278515, "ctx":"SignalHandler","msg":"Shutting down Change Stream Expired Pre-images Remover thread"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"QUERY", "id":6278516, "ctx":"SignalHandler","msg":"Finished shutting down Change Stream Expired Pre-images Remover thread"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"CONTROL", "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for shutdown"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"CONTROL", "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engine"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"STORAGE", "id":22320, "ctx":"SignalHandler","msg":"Shutting down journal flusher thread"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"STORAGE", "id":22321, "ctx":"SignalHandler","msg":"Finished shutting down journal flusher thread"*** ***"t":***"$date":"2025-09-11T09:29:32.137+00:00"***,"s":"I", "c":"STORAGE", "id":22322, "ctx":"SignalHandler","msg":"Shutting down checkpoint thread"*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":22323, "ctx":"SignalHandler","msg":"Finished shutting down checkpoint thread"*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":22261, "ctx":"SignalHandler","msg":"Timestamp monitor shutting down"*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":20282, "ctx":"SignalHandler","msg":"Deregistering all the collections"*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":22372, "ctx":"OplogVisibilityThread","msg":"Oplog visibility thread shutting down."*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":22317, "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting down"*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":22318, "ctx":"SignalHandler","msg":"Shutting down session sweeper thread"*** ***"t":***"$date":"2025-09-11T09:29:32.138+00:00"***,"s":"I", "c":"STORAGE", "id":22319, "ctx":"SignalHandler","msg":"Finished shutting down session sweeper thread"*** ***"t":***"$date":"2025-09-11T09:29:32.141+00:00"***,"s":"I", "c":"STORAGE", "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":***"closeConfig":"leak_memory=true,"*** ***"t":***"$date":"2025-09-11T09:29:32.143+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757582972,"ts_usec":143851,"thread":"19:0x7f83c161b640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown rollback to stable has successfully finished and ran for 1 milliseconds"*** ***"t":***"$date":"2025-09-11T09:29:32.227+00:00"***,"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757582972,"ts_usec":227088,"thread":"19:0x7f83c161b640","session_name":"close_ckpt","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 5265, snapshot max: 5265 snapshot count: 0, oldest timestamp: (1757582716, 2) , meta checkpoint timestamp: (1757582971, 2) base write gen: 1"*** ***"t":***"$date":"2025-09-11T09:29:32.296+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757582972,"ts_usec":296227,"thread":"19:0x7f83c161b640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown checkpoint has successfully finished and ran for 152 milliseconds"*** ***"t":***"$date":"2025-09-11T09:29:32.296+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757582972,"ts_usec":296397,"thread":"19:0x7f83c161b640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown was completed successfully and took 154ms, including 1ms for the rollback to stable, and 152ms for the checkpoint."*** ***"t":***"$date":"2025-09-11T09:29:32.326+00:00"***,"s":"I", "c":"STORAGE", "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":***"durationMillis":185*** ***"t":***"$date":"2025-09-11T09:29:32.326+00:00"***,"s":"I", "c":"STORAGE", "id":22279, "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."*** ***"t":***"$date":"2025-09-11T09:29:32.326+00:00"***,"s":"I", "c":"-", "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"*** ***"t":***"$date":"2025-09-11T09:29:32.326+00:00"***,"s":"I", "c":"FTDC", "id":20626, "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"*** ***"t":***"$date":"2025-09-11T09:29:32.331+00:00"***,"s":"I", "c":"CONTROL", "id":20565, "ctx":"SignalHandler","msg":"Now exiting"*** ***"t":***"$date":"2025-09-11T09:29:32.331+00:00"***,"s":"I", "c":"CONTROL", "id":8423404, "ctx":"SignalHandler","msg":"mongod shutdown complete","attr":***"Summary of time elapsed":***"Statistics":***"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"0 ms","Time spent in quiesce mode":"15013 ms","Shut down FLE Crud subsystem":"1 ms","Shut down MirrorMaestro":"0 ms","Shut down WaitForMajorityService":"2 ms","Shut down the logical session cache":"0 ms","Shut down the Query Analysis Sampler":"0 ms","Shut down the transport layer":"1 ms","Shut down the global connection pool":"0 ms","Shut down the flow control ticket holder":"0 ms","Shut down the replica set node executor":"0 ms","Shut down the replica set aware services":"1 ms","Shut down replication":"0 ms","Shut down initial syncer":"0 ms","Shut down external state":"809 ms","Shut down replication executor":"0 ms","Join replication executor":"5 ms","Kill all operations for shutdown":"0 ms","Shut down all tenant migration access blockers on global shutdown":"0 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"0 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"0 ms","Shut down the logical time validator":"0 ms","Shut down the migration util executor":"1 ms","Shut down the health log":"0 ms","Shut down the TTL monitor":"0 ms","Shut down expired pre-images and documents removers":"0 ms","Shut down the storage engine":"189 ms","Wait for the oplog cap maintainer thread to stop":"0 ms","Shut down full-time data capture":"0 ms","shutdownTask total elapsed time":"16027 ms"*** ***"t":***"$date":"2025-09-11T09:29:32.331+00:00"***,"s":"I", "c":"CONTROL", "id":23138, "ctx":"SignalHandler","msg":"Shutting down","attr":***"exitCode":0*** ==> /data/mongodb/logs/mongodb.log.2025-09-11T10-11-21 <== ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"CONTROL", "id":4784928, "ctx":"SignalHandler","msg":"Shutting down the TTL monitor"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"INDEX", "id":3684100, "ctx":"SignalHandler","msg":"Shutting down TTL collection monitor thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"INDEX", "id":3684101, "ctx":"SignalHandler","msg":"Finished shutting down TTL collection monitor thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"CONTROL", "id":6278511, "ctx":"SignalHandler","msg":"Shutting down the Change Stream Expired Pre-images Remover"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"QUERY", "id":6278515, "ctx":"SignalHandler","msg":"Shutting down Change Stream Expired Pre-images Remover thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"QUERY", "id":6278516, "ctx":"SignalHandler","msg":"Finished shutting down Change Stream Expired Pre-images Remover thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"CONTROL", "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for shutdown"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"CONTROL", "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engine"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"STORAGE", "id":22320, "ctx":"SignalHandler","msg":"Shutting down journal flusher thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"STORAGE", "id":22321, "ctx":"SignalHandler","msg":"Finished shutting down journal flusher thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"STORAGE", "id":22322, "ctx":"SignalHandler","msg":"Shutting down checkpoint thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"STORAGE", "id":22323, "ctx":"SignalHandler","msg":"Finished shutting down checkpoint thread"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"STORAGE", "id":22261, "ctx":"SignalHandler","msg":"Timestamp monitor shutting down"*** ***"t":***"$date":"2025-09-11T10:11:13.629+00:00"***,"s":"I", "c":"STORAGE", "id":20282, "ctx":"SignalHandler","msg":"Deregistering all the collections"*** ***"t":***"$date":"2025-09-11T10:11:13.630+00:00"***,"s":"I", "c":"STORAGE", "id":22372, "ctx":"OplogVisibilityThread","msg":"Oplog visibility thread shutting down."*** ***"t":***"$date":"2025-09-11T10:11:13.630+00:00"***,"s":"I", "c":"STORAGE", "id":22317, "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting down"*** ***"t":***"$date":"2025-09-11T10:11:13.630+00:00"***,"s":"I", "c":"STORAGE", "id":22318, "ctx":"SignalHandler","msg":"Shutting down session sweeper thread"*** ***"t":***"$date":"2025-09-11T10:11:13.630+00:00"***,"s":"I", "c":"STORAGE", "id":22319, "ctx":"SignalHandler","msg":"Finished shutting down session sweeper thread"*** ***"t":***"$date":"2025-09-11T10:11:13.630+00:00"***,"s":"I", "c":"STORAGE", "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":***"closeConfig":"leak_memory=true,"*** ***"t":***"$date":"2025-09-11T10:11:13.726+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757585473,"ts_usec":725979,"thread":"18:0x7f5eb97dd640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown rollback to stable has successfully finished and ran for 94 milliseconds"*** ***"t":***"$date":"2025-09-11T10:11:13.730+00:00"***,"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757585473,"ts_usec":730003,"thread":"18:0x7f5eb97dd640","session_name":"close_ckpt","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 7742, snapshot max: 7742 snapshot count: 0, oldest timestamp: (1757585102, 4) , meta checkpoint timestamp: (1757585402, 4) base write gen: 62"*** ***"t":***"$date":"2025-09-11T10:11:13.839+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757585473,"ts_usec":839310,"thread":"18:0x7f5eb97dd640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown checkpoint has successfully finished and ran for 113 milliseconds"*** ***"t":***"$date":"2025-09-11T10:11:13.839+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757585473,"ts_usec":839495,"thread":"18:0x7f5eb97dd640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown was completed successfully and took 208ms, including 94ms for the rollback to stable, and 113ms for the checkpoint."*** ***"t":***"$date":"2025-09-11T10:11:13.871+00:00"***,"s":"I", "c":"STORAGE", "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":***"durationMillis":241*** ***"t":***"$date":"2025-09-11T10:11:13.871+00:00"***,"s":"I", "c":"STORAGE", "id":22279, "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."*** ***"t":***"$date":"2025-09-11T10:11:13.871+00:00"***,"s":"I", "c":"-", "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"*** ***"t":***"$date":"2025-09-11T10:11:13.871+00:00"***,"s":"I", "c":"FTDC", "id":20626, "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"*** ***"t":***"$date":"2025-09-11T10:11:13.876+00:00"***,"s":"I", "c":"CONTROL", "id":20565, "ctx":"SignalHandler","msg":"Now exiting"*** ***"t":***"$date":"2025-09-11T10:11:13.925+00:00"***,"s":"I", "c":"CONTROL", "id":8423404, "ctx":"SignalHandler","msg":"mongod shutdown complete","attr":***"Summary of time elapsed":***"Statistics":***"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"0 ms","Time spent in quiesce mode":"15012 ms","Shut down FLE Crud subsystem":"0 ms","Shut down MirrorMaestro":"1 ms","Shut down WaitForMajorityService":"0 ms","Shut down the logical session cache":"0 ms","Shut down the Query Analysis Sampler":"1 ms","Shut down the transport layer":"0 ms","Shut down the global connection pool":"0 ms","Shut down the flow control ticket holder":"0 ms","Shut down the replica set node executor":"1 ms","Shut down the replica set aware services":"1 ms","Shut down replication":"0 ms","Shut down external state":"136 ms","Shut down replication executor":"1 ms","Join replication executor":"0 ms","Kill all operations for shutdown":"0 ms","Shut down all tenant migration access blockers on global shutdown":"0 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"0 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"0 ms","Shut down the logical time validator":"0 ms","Shut down the migration util executor":"1 ms","Shut down the health log":"0 ms","Shut down the TTL monitor":"0 ms","Shut down expired pre-images and documents removers":"0 ms","Shut down the storage engine":"242 ms","Wait for the oplog cap maintainer thread to stop":"0 ms","Shut down full-time data capture":"0 ms","shutdownTask total elapsed time":"15451 ms"*** ***"t":***"$date":"2025-09-11T10:11:13.925+00:00"***,"s":"I", "c":"CONTROL", "id":23138, "ctx":"SignalHandler","msg":"Shutting down","attr":***"exitCode":0*** ==> /data/mongodb/logs/mongodb.log.2025-09-11T10-23-40 <== ***"t":***"$date":"2025-09-11T10:20:47.129+00:00"***,"s":"I", "c":"CONTROL", "id":4784928, "ctx":"SignalHandler","msg":"Shutting down the TTL monitor"*** ***"t":***"$date":"2025-09-11T10:20:47.129+00:00"***,"s":"I", "c":"INDEX", "id":3684100, "ctx":"SignalHandler","msg":"Shutting down TTL collection monitor thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"INDEX", "id":3684101, "ctx":"SignalHandler","msg":"Finished shutting down TTL collection monitor thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"CONTROL", "id":6278511, "ctx":"SignalHandler","msg":"Shutting down the Change Stream Expired Pre-images Remover"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"QUERY", "id":6278515, "ctx":"SignalHandler","msg":"Shutting down Change Stream Expired Pre-images Remover thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"QUERY", "id":6278516, "ctx":"SignalHandler","msg":"Finished shutting down Change Stream Expired Pre-images Remover thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"CONTROL", "id":4784929, "ctx":"SignalHandler","msg":"Acquiring the global lock for shutdown"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"CONTROL", "id":4784930, "ctx":"SignalHandler","msg":"Shutting down the storage engine"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22320, "ctx":"SignalHandler","msg":"Shutting down journal flusher thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22321, "ctx":"SignalHandler","msg":"Finished shutting down journal flusher thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22322, "ctx":"SignalHandler","msg":"Shutting down checkpoint thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22323, "ctx":"SignalHandler","msg":"Finished shutting down checkpoint thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22261, "ctx":"SignalHandler","msg":"Timestamp monitor shutting down"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":20282, "ctx":"SignalHandler","msg":"Deregistering all the collections"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22372, "ctx":"OplogVisibilityThread","msg":"Oplog visibility thread shutting down."*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22317, "ctx":"SignalHandler","msg":"WiredTigerKVEngine shutting down"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22318, "ctx":"SignalHandler","msg":"Shutting down session sweeper thread"*** ***"t":***"$date":"2025-09-11T10:20:47.130+00:00"***,"s":"I", "c":"STORAGE", "id":22319, "ctx":"SignalHandler","msg":"Finished shutting down session sweeper thread"*** ***"t":***"$date":"2025-09-11T10:20:47.132+00:00"***,"s":"I", "c":"STORAGE", "id":4795902, "ctx":"SignalHandler","msg":"Closing WiredTiger","attr":***"closeConfig":"leak_memory=true,"*** ***"t":***"$date":"2025-09-11T10:20:47.135+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757586047,"ts_usec":134985,"thread":"20:0x7f06cb577640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown rollback to stable has successfully finished and ran for 2 milliseconds"*** ***"t":***"$date":"2025-09-11T10:20:47.138+00:00"***,"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757586047,"ts_usec":138424,"thread":"20:0x7f06cb577640","session_name":"close_ckpt","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 1177, snapshot max: 1177 snapshot count: 0, oldest timestamp: (1757585744, 2) , meta checkpoint timestamp: (1757586044, 2) base write gen: 538"*** ***"t":***"$date":"2025-09-11T10:20:47.216+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757586047,"ts_usec":216403,"thread":"20:0x7f06cb577640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown checkpoint has successfully finished and ran for 81 milliseconds"*** ***"t":***"$date":"2025-09-11T10:20:47.216+00:00"***,"s":"I", "c":"WTRECOV", "id":22430, "ctx":"SignalHandler","msg":"WiredTiger message","attr":***"message":***"ts_sec":1757586047,"ts_usec":216598,"thread":"20:0x7f06cb577640","session_name":"WT_CONNECTION.close","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG_1","verbose_level_id":1,"msg":"shutdown was completed successfully and took 83ms, including 2ms for the rollback to stable, and 81ms for the checkpoint."*** ***"t":***"$date":"2025-09-11T10:20:47.248+00:00"***,"s":"I", "c":"STORAGE", "id":4795901, "ctx":"SignalHandler","msg":"WiredTiger closed","attr":***"durationMillis":116*** ***"t":***"$date":"2025-09-11T10:20:47.249+00:00"***,"s":"I", "c":"STORAGE", "id":22279, "ctx":"SignalHandler","msg":"shutdown: removing fs lock..."*** ***"t":***"$date":"2025-09-11T10:20:47.249+00:00"***,"s":"I", "c":"-", "id":4784931, "ctx":"SignalHandler","msg":"Dropping the scope cache for shutdown"*** ***"t":***"$date":"2025-09-11T10:20:47.249+00:00"***,"s":"I", "c":"FTDC", "id":20626, "ctx":"SignalHandler","msg":"Shutting down full-time diagnostic data capture"*** ***"t":***"$date":"2025-09-11T10:20:47.253+00:00"***,"s":"I", "c":"CONTROL", "id":20565, "ctx":"SignalHandler","msg":"Now exiting"*** ***"t":***"$date":"2025-09-11T10:20:47.253+00:00"***,"s":"I", "c":"CONTROL", "id":8423404, "ctx":"SignalHandler","msg":"mongod shutdown complete","attr":***"Summary of time elapsed":***"Statistics":***"Enter terminal shutdown":"0 ms","Step down the replication coordinator for shutdown":"1 ms","Time spent in quiesce mode":"15008 ms","Shut down FLE Crud subsystem":"1 ms","Shut down MirrorMaestro":"1 ms","Shut down WaitForMajorityService":"0 ms","Shut down the logical session cache":"0 ms","Shut down the Query Analysis Sampler":"1 ms","Shut down the transport layer":"0 ms","Shut down the global connection pool":"0 ms","Shut down the flow control ticket holder":"0 ms","Shut down the replica set node executor":"0 ms","Shut down the replica set aware services":"0 ms","Shut down replication":"0 ms","Shut down external state":"307 ms","Shut down replication executor":"0 ms","Join replication executor":"1 ms","Kill all operations for shutdown":"0 ms","Shut down all tenant migration access blockers on global shutdown":"0 ms","Shut down all open transactions":"0 ms","Acquire the RSTL for shutdown":"0 ms","Shut down the IndexBuildsCoordinator and wait for index builds to finish":"0 ms","Shut down the replica set monitor":"0 ms","Shut down the logical time validator":"0 ms","Shut down the migration util executor":"0 ms","Shut down the health log":"0 ms","Shut down the TTL monitor":"1 ms","Shut down expired pre-images and documents removers":"0 ms","Shut down the storage engine":"119 ms","Wait for the oplog cap maintainer thread to stop":"0 ms","Shut down full-time data capture":"0 ms","shutdownTask total elapsed time":"15445 ms"*** ***"t":***"$date":"2025-09-11T10:20:47.253+00:00"***,"s":"I", "c":"CONTROL", "id":23138, "ctx":"SignalHandler","msg":"Shutting down","attr":***"exitCode":0*** delete cluster mongodb-ejjbon `kbcli cluster delete mongodb-ejjbon --auto-approve --namespace ns-vuvkk ` Cluster mongodb-ejjbon deleted pod_info:mongodb-ejjbon-mongodb-0 4/4 Running 0 25m mongodb-ejjbon-mongodb-1 4/4 Running 1 (32m ago) 45m mongodb-ejjbon-mongodb-2 4/4 Running 0 53m No resources found in ns-vuvkk namespace. delete cluster pod done No resources found in ns-vuvkk namespace. check cluster resource non-exist OK: pvc No resources found in ns-vuvkk namespace. delete cluster done No resources found in ns-vuvkk namespace. No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. check resource cm non exists No resources found in ns-vuvkk namespace. Mongodb Test Suite All Done! Test Engine: mongodb Test Type: 6 --------------------------------------Mongodb (Topology = replicaset Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=mongodb-1.0.1;ComponentVersion=mongodb;ServiceVersion=7.0.18;]|[Description=Create a cluster with the specified component definition mongodb-1.0.1 and component version mongodb and service version 7.0.18] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=ozxfv]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=ozxfv;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=mongodb]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=mongodb]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[HorizontalScaling Out]|[ComponentName=mongodb]|[Description=HorizontalScaling Out the cluster specify component mongodb] [PASSED]|[HorizontalScaling In]|[ComponentName=mongodb]|[Description=HorizontalScaling In the cluster specify component mongodb] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=mongodb]|[Description=VolumeExpansion the cluster specify component mongodb] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[VerticalScaling]|[ComponentName=mongodb]|[Description=VerticalScaling the cluster specify component mongodb] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[RebuildInstance]|[ComponentName=mongodb]|[Description=Rebuild the cluster instance specify component mongodb] [PASSED]|[HscaleOfflineInstances]|[ComponentName=mongodb]|[Description=Hscale the cluster instances offline specify component mongodb] [PASSED]|[HscaleOnlineInstances]|[ComponentName=mongodb]|[Description=Hscale the cluster instances online specify component mongodb] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=mongodb]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mongodb]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[SwitchOver]|[ComponentName=mongodb]|[Description=SwitchOver the cluster specify component mongodb] [PASSED]|[Failover]|[HA=Pod Kill;ComponentName=mongodb]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [WARNING]|[CheckFailover]|[FailoverType=networkpartition]|[Description=-] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=mongodb]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=mongodb]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=mongodb]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=mongodb]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=datafile]|[Description=The cluster datafile Backup] [PASSED]|[Restore]|[BackupMethod=datafile]|[Description=The cluster datafile Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=datafile]|[Description=Delete the datafile restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=mongodb]|[Description=Rebuild the cluster instance specify component mongodb] [PASSED]|[Backup]|[BackupMethod=dump]|[Description=The cluster dump Backup] [PASSED]|[Restore]|[BackupMethod=dump]|[Description=The cluster dump Restore] [PASSED]|[Check Data]|[BackupMethod=dump]|[Description=Check the cluster data restore via dump] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=dump]|[Description=Delete the dump restore cluster] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Backup]|[BackupMethod=pbm-physical]|[Description=The cluster pbm-physical Backup] [PASSED]|[Restore]|[BackupMethod=pbm-physical]|[Description=The cluster pbm-physical Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=pbm-physical]|[Description=Delete the pbm-physical restore cluster] [PASSED]|[Backup]|[Schedule=true;BackupMethod=pbm-physical]|[Description=The cluster Schedule pbm-physical Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=pbm-physical]|[Description=The cluster Schedule pbm-physical Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=pbm-physical]|[Description=Delete the Schedule pbm-physical restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]