source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-azgwq ` `kubectl create namespace ns-azgwq` namespace/ns-azgwq created create namespace ns-azgwq done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 163M 0 --:--:-- --:--:-- --:--:-- 163M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:mongodb set component version set component version:mongodb set service versions:4.0.28,4.2.24,4.4.29,5.0.28,5.0.30,6.0.16,6.0.20,7.0.12,7.0.16,8.0.4 set service versions sorted:4.0.28,4.2.24,4.4.29,5.0.28,5.0.30,6.0.16,6.0.20,7.0.12,7.0.16,8.0.4 set mongodb component definition set mongodb component definition mongodb set replicas first:3,4.0.28|3,4.2.24|3,4.4.29|3,5.0.28|3,5.0.30|3,6.0.16|3,6.0.20|3,7.0.12|3,7.0.16|3,8.0.4 set replicas third:3,6.0.20 set replicas fourth:3,6.0.16 set minimum cmpv service version set minimum cmpv service version replicas:3,6.0.16 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Not found topology in cluster definition mongodb LIMIT_CPU:0.1 LIMIT_MEMORY:0.5 storage size: 3 No resources found in ns-azgwq namespace. termination_policy:Delete create 3 replica Delete mongodb cluster check cluster version check cluster definition check component definition set component definition by component version check cmpd by labels set component definition2: mongodb by component version:mongodb apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: mongodb-rxyipy namespace: ns-azgwq spec: terminationPolicy: Delete componentSpecs: - name: mongodb componentDef: mongodb serviceVersion: 6.0.16 replicas: 3 resources: requests: cpu: 100m memory: 0.5Gi limits: cpu: 100m memory: 0.5Gi updateStrategy: BestEffortParallel volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi `kubectl apply -f test_create_mongodb-rxyipy.yaml` cluster.apps.kubeblocks.io/mongodb-rxyipy created apply test_create_mongodb-rxyipy.yaml Success `rm -rf test_create_mongodb-rxyipy.yaml` check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Jun 19,2025 18:16 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: check pod mongodb-rxyipy-mongodb-0 container_name mongodb exist password 0M3A04b2z7kh69ij Container mongodb logs contain secret password:2025-06-19T10:17:39Z INFO MongoDB Create user: root, passwd: 0M3A04b2z7kh69ij, roles: map[db:admin role:root] describe cluster `kbcli cluster describe mongodb-rxyipy --namespace ns-azgwq ` Name: mongodb-rxyipy Created Time: Jun 19,2025 18:16 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-azgwq Running Delete Endpoints: COMPONENT MODE INTERNAL EXTERNAL mongodb ReadWrite mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 mongodb-rxyipy-mongodb-mongodb.ns-azgwq.svc.cluster.local:27017 mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME mongodb mongodb-rxyipy-mongodb-0 primary Running us-west-2a ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb mongodb-rxyipy-mongodb-1 secondary Running us-west-2a ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb mongodb-rxyipy-mongodb-2 secondary Running us-west-2a ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS mongodb false 100m / 100m 512Mi / 512Mi data:3Gi kb-default-sc Images: COMPONENT TYPE IMAGE mongodb docker.io/apecloud/mongo:6.0.16 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-azgwq mongodb-rxyipy `kbcli cluster label mongodb-rxyipy app.kubernetes.io/instance- --namespace ns-azgwq ` label "app.kubernetes.io/instance" not found. `kbcli cluster label mongodb-rxyipy app.kubernetes.io/instance=mongodb-rxyipy --namespace ns-azgwq ` `kbcli cluster label mongodb-rxyipy --list --namespace ns-azgwq ` NAME NAMESPACE LABELS mongodb-rxyipy ns-azgwq app.kubernetes.io/instance=mongodb-rxyipy label cluster app.kubernetes.io/instance=mongodb-rxyipy Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=mongodb-rxyipy --namespace ns-azgwq ` `kbcli cluster label mongodb-rxyipy --list --namespace ns-azgwq ` NAME NAMESPACE LABELS mongodb-rxyipy ns-azgwq app.kubernetes.io/instance=mongodb-rxyipy case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label mongodb-rxyipy case.name=kbcli.test2 --overwrite --namespace ns-azgwq ` `kbcli cluster label mongodb-rxyipy --list --namespace ns-azgwq ` NAME NAMESPACE LABELS mongodb-rxyipy ns-azgwq app.kubernetes.io/instance=mongodb-rxyipy case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label mongodb-rxyipy case.name- --namespace ns-azgwq ` `kbcli cluster label mongodb-rxyipy --list --namespace ns-azgwq ` NAME NAMESPACE LABELS mongodb-rxyipy ns-azgwq app.kubernetes.io/instance=mongodb-rxyipy delete cluster label case.name Success list-accounts on characterType mongodb is not supported yet cluster connect `echo 'rs.status()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Current Mongosh Log ID: 6853e4729e9a2ba2205e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:17:32.432+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:17:32.432+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> *** set: 'mongodb-rxyipy-mongodb', date: ISODate('2025-06-19T10:21:09.711Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-06-19T10:21:05.311Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-06-19T10:21:05.311Z'), lastDurableWallTime: ISODate('2025-06-19T10:21:05.311Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1750328425, i: 1 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-06-19T10:17:34.512Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1750328254, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1750328254, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 1, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), newTermStartDate: ISODate('2025-06-19T10:17:34.727Z'), wMajorityWriteAvailabilityDate: ISODate('2025-06-19T10:17:34.919Z') ***, members: [ *** _id: 0, name: 'mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 229, optime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T10:21:05.000Z'), lastAppliedWallTime: ISODate('2025-06-19T10:21:05.311Z'), lastDurableWallTime: ISODate('2025-06-19T10:21:05.311Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp(*** t: 1750328254, i: 2 ***), electionDate: ISODate('2025-06-19T10:17:34.000Z'), configVersion: 5, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 178, optime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T10:21:05.000Z'), optimeDurableDate: ISODate('2025-06-19T10:21:05.000Z'), lastAppliedWallTime: ISODate('2025-06-19T10:21:05.311Z'), lastDurableWallTime: ISODate('2025-06-19T10:21:05.311Z'), lastHeartbeat: ISODate('2025-06-19T10:21:08.711Z'), lastHeartbeatRecv: ISODate('2025-06-19T10:21:07.712Z'), pingMs: Long('39'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 5, configTerm: 1 ***, *** _id: 2, name: 'mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 149, optime: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750328465, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T10:21:05.000Z'), optimeDurableDate: ISODate('2025-06-19T10:21:05.000Z'), lastAppliedWallTime: ISODate('2025-06-19T10:21:05.311Z'), lastDurableWallTime: ISODate('2025-06-19T10:21:05.311Z'), lastHeartbeat: ISODate('2025-06-19T10:21:07.611Z'), lastHeartbeatRecv: ISODate('2025-06-19T10:21:08.612Z'), pingMs: Long('37'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 1, infoMessage: '', configVersion: 5, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1750328465, i: 1 ***), signature: *** hash: Binary.createFromBase64('1riXQ3lmmbmSDd7BERHM4qEpl08=', 0), keyId: Long('7517602608194781190') *** ***, operationTime: Timestamp(*** t: 1750328465, i: 1 ***) *** mongodb-rxyipy-mongodb [primary] admin> connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-mongodb-rxyipy" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mongodb-rxyipy --namespace ns-azgwq ` Error from server (NotFound): pods "test-db-client-executionloop-mongodb-rxyipy" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-mongodb-rxyipy" not found `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-mongodb-rxyipy namespace: ns-azgwq spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local" - "--user" - "root" - "--password" - "0M3A04b2z7kh69ij" - "--port" - "27017" - "--dbtype" - "mongodb" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-mongodb-rxyipy.yaml` pod/test-db-client-executionloop-mongodb-rxyipy created apply test-db-client-executionloop-mongodb-rxyipy.yaml Success `rm -rf test-db-client-executionloop-mongodb-rxyipy.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 1/1 Running 0 59s check pod test-db-client-executionloop-mongodb-rxyipy status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-mongodb-rxyipy 0/1 Completed 0 65s check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done Inserted document: BsonObjectId***value=6853e4db2a5aa548d056173d*** 10:22:19.214 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 300, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328539, "i": 1***, "signature": ***"hash": ***"$binary": ***"base64": "U4PEgCMawFqDgK/rzmQOP0TnwIE=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4db2a5aa548d056173e"***, "value": "executions_loop_test_300"***]***' with request id 366 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:19.512 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 366 completed successfully in 298.02 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4db2a5aa548d056173e*** 10:22:19.513 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 301, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328539, "i": 2***, "signature": ***"hash": ***"$binary": ***"base64": "U4PEgCMawFqDgK/rzmQOP0TnwIE=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4db2a5aa548d056173f"***, "value": "executions_loop_test_301"***]***' with request id 367 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:19.618 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 367 completed successfully in 105.22 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4db2a5aa548d056173f*** [ 51s ] executions total: 306 successful: 301 failed: 5 disconnect: 1 10:22:19.620 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 302, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328539, "i": 3***, "signature": ***"hash": ***"$binary": ***"base64": "U4PEgCMawFqDgK/rzmQOP0TnwIE=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4db2a5aa548d0561740"***, "value": "executions_loop_test_302"***]***' with request id 368 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:19.912 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 368 completed successfully in 292.44 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4db2a5aa548d0561740*** 10:22:19.913 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 303, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328539, "i": 4***, "signature": ***"hash": ***"$binary": ***"base64": "U4PEgCMawFqDgK/rzmQOP0TnwIE=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4db2a5aa548d0561741"***, "value": "executions_loop_test_303"***]***' with request id 369 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:20.013 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 369 completed successfully in 99.68 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4db2a5aa548d0561741*** 10:22:20.014 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 304, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328539, "i": 5***, "signature": ***"hash": ***"$binary": ***"base64": "U4PEgCMawFqDgK/rzmQOP0TnwIE=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dc2a5aa548d0561742"***, "value": "executions_loop_test_304"***]***' with request id 370 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:20.124 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 370 completed successfully in 109.92 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dc2a5aa548d0561742*** 10:22:20.127 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 305, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328540, "i": 1***, "signature": ***"hash": ***"$binary": ***"base64": "Pjx+Y2cSb1gL8XYjoO+Ul/z0N1k=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dc2a5aa548d0561743"***, "value": "executions_loop_test_305"***]***' with request id 371 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:20.413 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 371 completed successfully in 286.94 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dc2a5aa548d0561743*** 10:22:20.414 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 306, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328540, "i": 2***, "signature": ***"hash": ***"$binary": ***"base64": "Pjx+Y2cSb1gL8XYjoO+Ul/z0N1k=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dc2a5aa548d0561744"***, "value": "executions_loop_test_306"***]***' with request id 372 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:20.612 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 372 completed successfully in 198.03 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dc2a5aa548d0561744*** 10:22:20.614 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 307, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328540, "i": 3***, "signature": ***"hash": ***"$binary": ***"base64": "Pjx+Y2cSb1gL8XYjoO+Ul/z0N1k=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dc2a5aa548d0561745"***, "value": "executions_loop_test_307"***]***' with request id 373 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:20.814 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 373 completed successfully in 200.18 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dc2a5aa548d0561745*** [ 52s ] executions total: 312 successful: 307 failed: 5 disconnect: 1 10:22:20.816 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 308, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328540, "i": 4***, "signature": ***"hash": ***"$binary": ***"base64": "Pjx+Y2cSb1gL8XYjoO+Ul/z0N1k=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dc2a5aa548d0561746"***, "value": "executions_loop_test_308"***]***' with request id 374 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:21.014 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 374 completed successfully in 198.12 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dc2a5aa548d0561746*** 10:22:21.015 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 309, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328540, "i": 5***, "signature": ***"hash": ***"$binary": ***"base64": "Pjx+Y2cSb1gL8XYjoO+Ul/z0N1k=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dd2a5aa548d0561747"***, "value": "executions_loop_test_309"***]***' with request id 375 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:21.212 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 375 completed successfully in 197.18 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dd2a5aa548d0561747*** 10:22:21.213 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 310, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328541, "i": 1***, "signature": ***"hash": ***"$binary": ***"base64": "zN3izz2NF3CQDa+sSWwGpDNLBuk=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dd2a5aa548d0561748"***, "value": "executions_loop_test_310"***]***' with request id 376 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:21.414 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 376 completed successfully in 200.39 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dd2a5aa548d0561748*** 10:22:21.416 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 311, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328541, "i": 2***, "signature": ***"hash": ***"$binary": ***"base64": "zN3izz2NF3CQDa+sSWwGpDNLBuk=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dd2a5aa548d0561749"***, "value": "executions_loop_test_311"***]***' with request id 377 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:21.627 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 377 completed successfully in 211.99 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dd2a5aa548d0561749*** 10:22:21.629 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 312, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328541, "i": 3***, "signature": ***"hash": ***"$binary": ***"base64": "zN3izz2NF3CQDa+sSWwGpDNLBuk=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dd2a5aa548d056174a"***, "value": "executions_loop_test_312"***]***' with request id 378 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:21.636 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 378 completed successfully in 7.53 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dd2a5aa548d056174a*** 10:22:21.638 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 313, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328541, "i": 4***, "signature": ***"hash": ***"$binary": ***"base64": "zN3izz2NF3CQDa+sSWwGpDNLBuk=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dd2a5aa548d056174b"***, "value": "executions_loop_test_313"***]***' with request id 379 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:21.819 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 379 completed successfully in 181.87 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dd2a5aa548d056174b*** [ 53s ] executions total: 318 successful: 313 failed: 5 disconnect: 1 10:22:21.821 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 314, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328541, "i": 5***, "signature": ***"hash": ***"$binary": ***"base64": "zN3izz2NF3CQDa+sSWwGpDNLBuk=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4dd2a5aa548d056174c"***, "value": "executions_loop_test_314"***]***' with request id 380 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:22.113 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 380 completed successfully in 292.55 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4dd2a5aa548d056174c*** 10:22:22.115 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 315, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328541, "i": 6***, "signature": ***"hash": ***"$binary": ***"base64": "zN3izz2NF3CQDa+sSWwGpDNLBuk=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4de2a5aa548d056174d"***, "value": "executions_loop_test_315"***]***' with request id 381 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:22.513 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 381 completed successfully in 398.32 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4de2a5aa548d056174d*** 10:22:22.514 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 316, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328542, "i": 1***, "signature": ***"hash": ***"$binary": ***"base64": "3VlviBcbqi4tPNO/11PGLJcDn4A=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4de2a5aa548d056174e"***, "value": "executions_loop_test_316"***]***' with request id 382 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:22.530 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 382 completed successfully in 15.68 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4de2a5aa548d056174e*** 10:22:22.531 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 317, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328542, "i": 2***, "signature": ***"hash": ***"$binary": ***"base64": "3VlviBcbqi4tPNO/11PGLJcDn4A=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4de2a5aa548d056174f"***, "value": "executions_loop_test_317"***]***' with request id 383 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:22.630 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 383 completed successfully in 99.02 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4de2a5aa548d056174f*** 10:22:22.632 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 318, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328542, "i": 3***, "signature": ***"hash": ***"$binary": ***"base64": "3VlviBcbqi4tPNO/11PGLJcDn4A=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4de2a5aa548d0561750"***, "value": "executions_loop_test_318"***]***' with request id 384 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:22.820 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 384 completed successfully in 188.58 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4de2a5aa548d0561750*** [ 54s ] executions total: 323 successful: 318 failed: 5 disconnect: 1 10:22:22.822 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 319, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328542, "i": 4***, "signature": ***"hash": ***"$binary": ***"base64": "3VlviBcbqi4tPNO/11PGLJcDn4A=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4de2a5aa548d0561751"***, "value": "executions_loop_test_319"***]***' with request id 385 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:22.918 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 385 completed successfully in 96.14 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4de2a5aa548d0561751*** 10:22:22.920 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 320, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328542, "i": 5***, "signature": ***"hash": ***"$binary": ***"base64": "3VlviBcbqi4tPNO/11PGLJcDn4A=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4de2a5aa548d0561752"***, "value": "executions_loop_test_320"***]***' with request id 386 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:23.212 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 386 completed successfully in 292.72 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4de2a5aa548d0561752*** 10:22:23.214 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 321, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328542, "i": 6***, "signature": ***"hash": ***"$binary": ***"base64": "3VlviBcbqi4tPNO/11PGLJcDn4A=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4df2a5aa548d0561753"***, "value": "executions_loop_test_321"***]***' with request id 387 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:23.516 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 387 completed successfully in 302.40 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4df2a5aa548d0561753*** 10:22:23.518 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"insert": "executions_loop_table", "ordered": true, "txnNumber": 322, "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328543, "i": 1***, "signature": ***"hash": ***"$binary": ***"base64": "C4EciHY+a8Ac1xwWPNlb3tfz8SM=", "subType": "00"***, "keyId": 7517602608194781190***, "lsid": ***"id": ***"$binary": ***"base64": "QmSAIWuhS/qCGW+1tHOzYQ==", "subType": "04"***, "documents": [***"_id": ***"$oid": "6853e4df2a5aa548d0561754"***, "value": "executions_loop_test_322"***]***' with request id 388 to database admin on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:23.629 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 388 completed successfully in 111.71 ms on connection [connectionId***localValue:18, serverValue:3286***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 Inserted document: BsonObjectId***value=6853e4df2a5aa548d0561754*** [ 60s ] executions total: 327 successful: 322 failed: 5 disconnect: 1 10:22:23.632 [main] DEBUG org.mongodb.driver.protocol.command -- Sending command '***"endSessions": [***"id": ***"$binary": ***"base64": "PINJ3F0FSOmHuU5T/dwmyw==", "subType": "04"***], "$db": "admin", "$clusterTime": ***"clusterTime": ***"$timestamp": ***"t": 1750328533, "i": 5***, "signature": ***"hash": ***"$binary": ***"base64": "9GOb8Xn9VDIq8KCzPx5ty3tGaH0=", "subType": "00"***, "keyId": 7517602608194781190***, "$readPreference": ***"mode": "primaryPreferred"***' with request id 389 to database admin on connection [connectionId***localValue:3, serverValue:1838***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:23.633 [main] DEBUG org.mongodb.driver.protocol.command -- Execution of command with request id 389 completed successfully in 0.94 ms on connection [connectionId***localValue:3, serverValue:1838***] to server mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 10:22:23.635 [main] DEBUG org.mongodb.driver.connection -- Closed connection [connectionId***localValue:3, serverValue:1838***] to mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local:27017 because the pool has been closed. 10:22:23.635 [main] DEBUG org.mongodb.driver.connection -- Closing connection connectionId***localValue:3, serverValue:1838*** 10:22:23.636 [main] DEBUG org.mongodb.driver.connection -- Closing connection connectionId***localValue:2, serverValue:2063*** 10:22:23.638 [main] DEBUG org.mongodb.driver.connection -- Closing connection connectionId***localValue:1, serverValue:3272*** Test Result: Total Executions: 327 Successful Executions: 322 Failed Executions: 5 Disconnection Counts: 1 Connection Information: Database Type: mongodb Host: mongodb-rxyipy-mongodb.ns-azgwq.svc.cluster.local Port: 27017 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 322 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-mongodb-rxyipy --namespace ns-azgwq ` pod/test-db-client-executionloop-mongodb-rxyipy patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-mongodb-rxyipy" force deleted `echo "db.col.remove(***) ; db.col.insertOne(***a:'egbjb'***)" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e51f63ea60a2f55e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:17:32.432+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:17:32.432+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> DeprecationWarning: Collection.remove() is deprecated. Use deleteOne, deleteMany, findOneAndDelete, or bulkWrite. *** acknowledged: true, insertedId: ObjectId('6853e54063ea60a2f55e739c') *** mongodb-rxyipy-mongodb [primary] admin> add consistent data egbjb Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.remove(***) ; db.col.insertOne(***a:'egbjb'***)\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin" | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e56125172d1f915e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T10:18:09.331+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:18:09.332+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> DeprecationWarning: Collection.remove() is deprecated. Use deleteOne, deleteMany, findOneAndDelete, or bulkWrite. Uncaught MongoServerError[NotWritablePrimary]: not primary admin> check add consistent data readonly Success cmpv upgrade service version:3,4.0.28|3,4.2.24|3,4.4.29|3,5.0.28|3,5.0.30|3,6.0.16|3,6.0.20|3,7.0.12|3,7.0.16|3,8.0.4 set latest cmpv service version latest service version:6.0.20 cmpv service version upgrade and downgrade upgrade from:6.0.16 to service version:6.0.20 cluster upgrade apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-rxyipy-upgrade-cmpv- namespace: ns-azgwq spec: clusterName: mongodb-rxyipy upgrade: components: - componentName: mongodb serviceVersion: 6.0.20 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-rxyipy.yaml` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-upgrade-cmpv-d2ckk created create test_ops_cluster_mongodb-rxyipy.yaml Success `rm -rf test_ops_cluster_mongodb-rxyipy.yaml` check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-upgrade-cmpv-d2ckk ns-azgwq Upgrade mongodb-rxyipy mongodb Running 0/3 Jun 19,2025 18:25 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-upgrade-cmpv-d2ckk ns-azgwq Upgrade mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:25 UTC+0800 check ops status done ops_status:mongodb-rxyipy-upgrade-cmpv-d2ckk ns-azgwq Upgrade mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:25 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-upgrade-cmpv-d2ckk --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-upgrade-cmpv-d2ckk patched `kbcli cluster delete-ops --name mongodb-rxyipy-upgrade-cmpv-d2ckk --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-upgrade-cmpv-d2ckk deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e653ab75ce90e1544ca6 Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.8 Using MongoDB: 6.0.20 Using Mongosh: 2.3.8 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:25:53.433+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:25:53.433+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e6848bc767e99e544ca6 Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.8 Using MongoDB: 6.0.20 Using Mongosh: 2.3.8 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T10:26:41.530+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:26:41.530+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success downgrade from:6.0.20 to service version:6.0.16 cluster upgrade apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-rxyipy-upgrade-cmpv- namespace: ns-azgwq spec: clusterName: mongodb-rxyipy upgrade: components: - componentName: mongodb serviceVersion: 6.0.16 type: Upgrade check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-rxyipy.yaml` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-upgrade-cmpv-jhlgf created create test_ops_cluster_mongodb-rxyipy.yaml Success `rm -rf test_ops_cluster_mongodb-rxyipy.yaml` check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-upgrade-cmpv-jhlgf ns-azgwq Upgrade mongodb-rxyipy Pending -/- Jun 19,2025 18:30 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-upgrade-cmpv-jhlgf ns-azgwq Upgrade mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:30 UTC+0800 check ops status done ops_status:mongodb-rxyipy-upgrade-cmpv-jhlgf ns-azgwq Upgrade mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-upgrade-cmpv-jhlgf --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-upgrade-cmpv-jhlgf patched `kbcli cluster delete-ops --name mongodb-rxyipy-upgrade-cmpv-jhlgf --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-upgrade-cmpv-jhlgf deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e7824ecbfd17ca5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:31:51.117+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:31:51.117+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e7b598bd5749c75e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T10:31:15.030+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:31:15.030+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-rxyipy namespace. `kbcli cluster hscale mongodb-rxyipy --auto-approve --force=true --components mongodb --replicas 4 --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-horizontalscaling-mdt2j created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-horizontalscaling-mdt2j -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-horizontalscaling-mdt2j ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 18:35 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 mongodb-rxyipy-mongodb-3 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 18:35 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 mongodb-rxyipy-mongodb-3 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done No resources found in mongodb-rxyipy namespace. check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-horizontalscaling-mdt2j ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 18:35 UTC+0800 check ops status done ops_status:mongodb-rxyipy-horizontalscaling-mdt2j ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 18:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-horizontalscaling-mdt2j --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-horizontalscaling-mdt2j patched `kbcli cluster delete-ops --name mongodb-rxyipy-horizontalscaling-mdt2j --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-horizontalscaling-mdt2j deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary), mongodb-rxyipy-mongodb-3(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e886c891d825355e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:31:51.117+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:31:51.117+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e8bd231caaa4e25e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:31:15.030+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:31:15.030+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-rxyipy namespace. `kbcli cluster hscale mongodb-rxyipy --auto-approve --force=true --components mongodb --replicas 3 --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-horizontalscaling-s4pl2 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-horizontalscaling-s4pl2 -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-horizontalscaling-s4pl2 ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 18:40 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 18:17 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-8-116.us-west-2.compute.internal/172.31.8.116 Jun 19,2025 18:18 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done No resources found in mongodb-rxyipy namespace. check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-horizontalscaling-s4pl2 ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 18:40 UTC+0800 check ops status done ops_status:mongodb-rxyipy-horizontalscaling-s4pl2 ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 18:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-horizontalscaling-s4pl2 --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-horizontalscaling-s4pl2 patched `kbcli cluster delete-ops --name mongodb-rxyipy-horizontalscaling-s4pl2 --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-horizontalscaling-s4pl2 deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e963a5bbc032665e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:31:51.117+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:31:51.117+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853e9940ad01623965e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:31:15.030+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:31:15.030+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster configure component_tmp: mongodb check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure mongodb-rxyipy --auto-approve --force=true --set systemLog.quiet=true --components mongodb --config-spec mongodb-config --config-file mongodb.conf --namespace ns-azgwq ` Will updated configure file meta: ConfigSpec: mongodb-config ConfigFile: mongodb.conf ComponentName: mongodb ClusterName: mongodb-rxyipy OpsRequest mongodb-rxyipy-reconfiguring-chhgx created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-reconfiguring-chhgx -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-reconfiguring-chhgx ns-azgwq Reconfiguring mongodb-rxyipy mongodb Running -/- Jun 19,2025 18:43 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:46 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:45 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:44 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-reconfiguring-chhgx ns-azgwq Reconfiguring mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:43 UTC+0800 check ops status done ops_status:mongodb-rxyipy-reconfiguring-chhgx ns-azgwq Reconfiguring mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-reconfiguring-chhgx --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-reconfiguring-chhgx patched `kbcli cluster delete-ops --name mongodb-rxyipy-reconfiguring-chhgx --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-reconfiguring-chhgx deleted check config variables Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: true configure:[systemLog.quiet] result actual:[true] equal expected:[true] `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853eb0c3074df7a995e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:44:55.186+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:44:55.186+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853eb3e5ea54fbec75e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T10:46:43.617+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:46:43.617+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-rxyipy" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-2 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-mongodb-rxyipy.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-mongodb-rxyipy created apply test-chaos-mesh-timeoffset-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-timeoffset-mongodb-rxyipy.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:46 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-15-106.us-west-2.compute.internal/172.31.15.106 Jun 19,2025 18:45 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:44 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-rxyipy" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-2 failover timeoffset Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ec46b7bbc9a0445e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:44:55.186+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:44:55.186+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ec784aff562ffa5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:46:43.617+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:46:43.617+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `mongosh ***mongodb-rxyipy-mongodb-mongodb.ns-azgwq.svc.cluster.local` Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg:Current Mongosh Log ID: 6853ed0e721a5b26b35e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb.ns-azgwq.svc.cluster.local/?directConnection=true&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:44:55.186+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:44:55.186+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: primary] test> connect headlessEndpoints Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart mongodb-rxyipy --auto-approve --force=true --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-restart-2htx5 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-restart-2htx5 -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-restart-2htx5 ns-azgwq Restart mongodb-rxyipy mongodb Running 0/3 Jun 19,2025 18:57 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:59 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 18:57 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-restart-2htx5 ns-azgwq Restart mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:57 UTC+0800 check ops status done ops_status:mongodb-rxyipy-restart-2htx5 ns-azgwq Restart mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 18:57 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-restart-2htx5 --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-restart-2htx5 patched `kbcli cluster delete-ops --name mongodb-rxyipy-restart-2htx5 --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-restart-2htx5 deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ee135b4d49a1d95e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:58:48.695+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:58:48.696+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ee49db622ad7615e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:00:31.487+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:00:31.487+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote mongodb-rxyipy --auto-approve --force=true --component mongodb --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-custom-lqsgr created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-custom-lqsgr -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-custom-lqsgr ns-azgwq Custom mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:03 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:59 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 18:57 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-custom-lqsgr ns-azgwq Custom mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 19:03 UTC+0800 check ops status done ops_status:mongodb-rxyipy-custom-lqsgr ns-azgwq Custom mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 19:03 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-custom-lqsgr --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-custom-lqsgr patched `kbcli cluster delete-ops --name mongodb-rxyipy-custom-lqsgr --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-custom-lqsgr deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853eee2bc9d6553485e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:00:31.487+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:00:31.487+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ef138da46aff5f5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:59:37.614+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:59:37.615+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success switchover pod:mongodb-rxyipy-mongodb-2 switchover success cluster update enable-all-logs `kbcli cluster update mongodb-rxyipy --enable-all-logs=true --namespace ns-azgwq ` error: name is required check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:59 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 18:57 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-rxyipy" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-2 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-mongodb-rxyipy.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-mongodb-rxyipy created apply test-chaos-mesh-fullcpu-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-fullcpu-mongodb-rxyipy.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 18:59 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-155.us-west-2.compute.internal/172.31.6.155 Jun 19,2025 18:57 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 100m / 100m 512Mi / 512Mi data:3Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 18:59 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-rxyipy" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-2 failover fullcpu Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f0b4b39f87c22b5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:00:31.487+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:00:31.487+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f0e794c5c98b395e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T10:58:48.695+00:00: You are running this process as the root user, which is not recommended 2025-06-19T10:58:48.696+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale mongodb-rxyipy --auto-approve --force=true --components mongodb --cpu 200m --memory 0.6Gi --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-verticalscaling-k4lfm created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-verticalscaling-k4lfm -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-verticalscaling-k4lfm ns-azgwq VerticalScaling mongodb-rxyipy mongodb Running 0/3 Jun 19,2025 19:15 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:15 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:3Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-verticalscaling-k4lfm ns-azgwq VerticalScaling mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 19:15 UTC+0800 check ops status done ops_status:mongodb-rxyipy-verticalscaling-k4lfm ns-azgwq VerticalScaling mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 19:15 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-verticalscaling-k4lfm --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-verticalscaling-k4lfm patched `kbcli cluster delete-ops --name mongodb-rxyipy-verticalscaling-k4lfm --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-verticalscaling-k4lfm deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f1ef874dfa014d5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:15:54.951+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:15:54.952+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f20523b0ed5c385e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=mongodb-rxyipy,apps.kubeblocks.io/component-name=mongodb,apps.kubeblocks.io/vct-name=data --namespace ns-azgwq ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in mongodb-rxyipy namespace. `kbcli cluster volume-expand mongodb-rxyipy --auto-approve --force=true --components mongodb --volume-claim-templates data --storage 5Gi --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-volumeexpansion-wdsr4 created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-volumeexpansion-wdsr4 -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-volumeexpansion-wdsr4 ns-azgwq VolumeExpansion mongodb-rxyipy mongodb Running 0/3 Jun 19,2025 19:18 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:15 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done No resources found in mongodb-rxyipy namespace. check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-volumeexpansion-wdsr4 ns-azgwq VolumeExpansion mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 19:18 UTC+0800 check ops status done ops_status:mongodb-rxyipy-volumeexpansion-wdsr4 ns-azgwq VolumeExpansion mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 19:18 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-volumeexpansion-wdsr4 --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-volumeexpansion-wdsr4 patched `kbcli cluster delete-ops --name mongodb-rxyipy-volumeexpansion-wdsr4 --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-volumeexpansion-wdsr4 deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f296ed1572c0725e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:15:54.951+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:15:54.952+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f2acbd9ccd3c975e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:16:32.026+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:16:32.027+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover delete pod:mongodb-rxyipy-mongodb-1 `kubectl delete pod mongodb-rxyipy-mongodb-1 --namespace ns-azgwq ` pod "mongodb-rxyipy-mongodb-1" deleted check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check failover pod name failover pod name:mongodb-rxyipy-mongodb-0 failover Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f30686ba806ff95e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:16:32.026+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:16:32.027+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f31d7995d53c9d5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:22:17.957+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:22:17.957+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-rxyipy" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-mongodb-rxyipy.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-mongodb-rxyipy created apply test-chaos-mesh-podfailure-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-podfailure-mongodb-rxyipy.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-rxyipy" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-1 failover podfailure Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f3d866e6bd741d5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:22:17.957+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:22:17.957+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f3ee4f1ad12c215e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-rxyipy" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-1 action: partition mode: all target: mode: all selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-mongodb-rxyipy.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-mongodb-rxyipy created apply test-chaos-mesh-networkpartition-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-networkpartition-mongodb-rxyipy.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-rxyipy" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-1 failover networkpartition Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f4a78debae3d615e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:22:17.957+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:22:17.957+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f4be78b07835da5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:25:46.926+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:25:46.926+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check failover pod name failover pod name:mongodb-rxyipy-mongodb-0 failover kill1 Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f517af0c3e5e035e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:25:46.926+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:25:46.926+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f52f6668e645dc5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-rxyipy" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-mongodb-rxyipy.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mongodb-rxyipy created apply test-chaos-mesh-oom-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-oom-mongodb-rxyipy.yaml` check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-mongodb-rxyipy" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-mongodb-rxyipy patched check failover pod name failover pod name:mongodb-rxyipy-mongodb-1 failover oom Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f5903ed1f9d3175e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:31:00.652+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:31:00.653+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f5a6a95911bd9d5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-rxyipy" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-1 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-mongodb-rxyipy.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-mongodb-rxyipy created apply test-chaos-mesh-dnserror-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-dnserror-mongodb-rxyipy.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-rxyipy" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-1 failover dnserror Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f659da14df5cb35e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:31:00.652+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:31:00.653+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f66fc97eaaacb05e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-rxyipy" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-1 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-mongodb-rxyipy.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-mongodb-rxyipy created apply test-chaos-mesh-dnsrandom-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-dnsrandom-mongodb-rxyipy.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-13-32.us-west-2.compute.internal/172.31.13.32 Jun 19,2025 19:21 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-rxyipy" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-1 failover dnsrandom Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f723129ed0131d5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:31:00.652+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:31:00.653+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f73a0e51a288dd5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:33:02.621+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:33:02.621+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-13-32.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-12-104.us-west-2.compute.internal success kubectl get pod mongodb-rxyipy-mongodb-2 --namespace ns-azgwq -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-2-181.us-west-2.compute.internal success kubectl drain ip-172-31-13-32.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-13-32.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-db2fj, kb-mqtcr/kb-addon-apecloud-otel-collector-jvh6n, kube-system/aws-node-k4wbp, kube-system/ebs-csi-node-85fqt, kube-system/kube-proxy-frgj5 evicting pod ns-dmaln/restore-preparedata-f3c6dabb-backup-ns-dmaln-apemysql-erd-9vkfd evicting pod ns-azgwq/mongodb-rxyipy-mongodb-1 pod/mongodb-rxyipy-mongodb-1 evicted pod/restore-preparedata-f3c6dabb-backup-ns-dmaln-apemysql-erd-9vkfd evicted node/ip-172-31-13-32.us-west-2.compute.internal drained kubectl uncordon ip-172-31-13-32.us-west-2.compute.internal node/ip-172-31-13-32.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check failover pod name failover pod name:mongodb-rxyipy-mongodb-0 failover drainnode Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f7b9277a29fcd95e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:33:02.621+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:33:02.621+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f7d00a3cb44bb15e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster update monitor KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done check addon prometheus enabled check addon prometheus enabled done check addon grafana enabled check addon grafana enabled done check addon apecloud-otel-collector enabled check addon apecloud-otel-collector enabled done kubectl patch cluster mongodb-rxyipy --namespace ns-azgwq --type json -p '[***"op": "replace", "path": "/spec/componentSpecs/0/disableExporter", "value": false***]' cluster.apps.kubeblocks.io/mongodb-rxyipy patched check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done create curl-jq pod mongodb-rxyipy-kb-curl `kubectl run --namespace kb-mqtcr mongodb-rxyipy-kb-curl --restart='Never' --image docker.io/apecloud/curl-jq:0.1.0 --command -- sleep infinity ` pod/mongodb-rxyipy-kb-curl created check pod status pod_status:NAME READY STATUS RESTARTS AGE mongodb-rxyipy-kb-curl 0/1 ContainerCreating 0 1s check pod mongodb-rxyipy-kb-curl status done `kubectl wait --for=condition=Ready pod mongodb-rxyipy-kb-curl --namespace kb-mqtcr` pod/mongodb-rxyipy-kb-curl condition met pod_status:NAME READY STATUS RESTARTS AGE mongodb-rxyipy-kb-curl 1/1 Running 0 6s kubectl exec -it mongodb-rxyipy-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)" `kubectl exec -it mongodb-rxyipy-kb-curl --namespace kb-mqtcr -- curl -G "http://kb-addon-prometheus-server:80/api/v1/query" -d "query=sum(k8s_pod_cpu_limit)"` Unable to use a TTY - input is not a terminal or the right kind of file % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 122 100 122 0 0 39127 0 --:--:-- --:--:-- --:--:-- 61000 monitor_result:***"status":"success","data":***"resultType":"vector","result":[***"metric":***,"value":[1750333463.659,"12.199999999999992"]***]*** k8s_pod_cpu_limit: 1750333463.659 monitor cluster Success `kubectl delete pod mongodb-rxyipy-kb-curl --namespace kb-mqtcr --force` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "mongodb-rxyipy-kb-curl" force deleted check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-rxyipy" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-mongodb-rxyipy.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-mongodb-rxyipy created apply test-chaos-mesh-networkbandwidthover-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-mongodb-rxyipy.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 19:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-rxyipy" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-2 failover networkbandwidthover Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f8d0da0a7e14c25e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f8e9133771ddab5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:41:42.317+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:41:42.317+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster rebulid instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-rxyipy-rebuildinstance- namespace: ns-azgwq spec: type: RebuildInstance clusterName: mongodb-rxyipy force: true rebuildFrom: - componentName: mongodb instances: - name: mongodb-rxyipy-mongodb-0 inPlace: true check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-rxyipy.yaml` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-rebuildinstance-qrm9j created create test_ops_cluster_mongodb-rxyipy.yaml Success `rm -rf test_ops_cluster_mongodb-rxyipy.yaml` check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy Creating -/- Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 19:48 UTC+0800 check ops status done ops_status:mongodb-rxyipy-rebuildinstance-qrm9j ns-azgwq RebuildInstance mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 19:48 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-rebuildinstance-qrm9j --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-rebuildinstance-qrm9j patched `kbcli cluster delete-ops --name mongodb-rxyipy-rebuildinstance-qrm9j --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-rebuildinstance-qrm9j deleted check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:48 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f9680e4f8dfeb35e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853f9815ef540a8e05e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T11:49:07.088+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:49:07.088+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-rxyipy" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-2 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-mongodb-rxyipy.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-mongodb-rxyipy created apply test-chaos-mesh-networkduplicate-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-networkduplicate-mongodb-rxyipy.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:48 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-rxyipy" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-2 failover networkduplicate Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fa3871a583b4795e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fa501768e4b68a5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:49:07.088+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:49:07.088+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-rxyipy" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-2 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-mongodb-rxyipy.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-mongodb-rxyipy created apply test-chaos-mesh-networkcorruptover-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-mongodb-rxyipy.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:48 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-1;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-rxyipy" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-1 failover networkcorruptover Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-1: out of mongodb-rxyipy-mongodb-1(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fb494822992b8a5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-1.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:41:42.317+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:41:42.317+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fb61cb19e2328f5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:49:07.088+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:49:07.088+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success skip cluster Upgrade test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-rxyipy" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-1 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-mongodb-rxyipy.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-mongodb-rxyipy created apply test-chaos-mesh-networklossover-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-networklossover-mongodb-rxyipy.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:48 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-rxyipy" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-2 failover networklossover Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fc189ddbe87bfe5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fc31a16cb598ba5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:41:42.317+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:41:42.317+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-rxyipy" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-rxyipy" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-mongodb-rxyipy namespace: ns-azgwq spec: selector: namespaces: - ns-azgwq labelSelectors: apps.kubeblocks.io/pod-name: mongodb-rxyipy-mongodb-2 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-mongodb-rxyipy.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-mongodb-rxyipy created apply test-chaos-mesh-networkdelay-mongodb-rxyipy.yaml Success `rm -rf test-chaos-mesh-networkdelay-mongodb-rxyipy.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 19:48 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-mongodb-rxyipy --namespace ns-azgwq ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-rxyipy" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-mongodb-rxyipy" not found check failover pod name failover pod name:mongodb-rxyipy-mongodb-2 failover networkdelay Success `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fcea2b22cc4e925e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fd01a0a87455db5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:49:07.088+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:49:07.088+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster hscale offline instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-rxyipy-hscaleoffinstance- labels: app.kubernetes.io/instance: mongodb-rxyipy app.kubernetes.io/managed-by: kubeblocks namespace: ns-azgwq spec: type: HorizontalScaling clusterName: mongodb-rxyipy force: true horizontalScaling: - componentName: mongodb scaleIn: onlineInstancesToOffline: - mongodb-rxyipy-mongodb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-rxyipy.yaml` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-hscaleoffinstance-w427b created create test_ops_cluster_mongodb-rxyipy.yaml Success `rm -rf test_ops_cluster_mongodb-rxyipy.yaml` check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-hscaleoffinstance-w427b ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Creating -/- Jun 19,2025 20:05 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-hscaleoffinstance-w427b ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 20:05 UTC+0800 check ops status done ops_status:mongodb-rxyipy-hscaleoffinstance-w427b ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 20:05 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-hscaleoffinstance-w427b --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-hscaleoffinstance-w427b patched `kbcli cluster delete-ops --name mongodb-rxyipy-hscaleoffinstance-w427b --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-hscaleoffinstance-w427b deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fd4dfd45c9600e5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fd6480bc289e445e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:41:42.317+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:41:42.317+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster hscale online instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-rxyipy-hscaleoninstance- labels: app.kubernetes.io/instance: mongodb-rxyipy app.kubernetes.io/managed-by: kubeblocks namespace: ns-azgwq spec: type: HorizontalScaling clusterName: mongodb-rxyipy force: true horizontalScaling: - componentName: mongodb scaleOut: offlineInstancesToOnline: - mongodb-rxyipy-mongodb-0 check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-rxyipy.yaml` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-hscaleoninstance-smbbn created create test_ops_cluster_mongodb-rxyipy.yaml Success `rm -rf test_ops_cluster_mongodb-rxyipy.yaml` check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-hscaleoninstance-smbbn ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Running 0/1 Jun 19,2025 20:07 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 20:07 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 19:41 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 19:16 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-hscaleoninstance-smbbn ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 20:07 UTC+0800 check ops status done ops_status:mongodb-rxyipy-hscaleoninstance-smbbn ns-azgwq HorizontalScaling mongodb-rxyipy mongodb Succeed 1/1 Jun 19,2025 20:07 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-hscaleoninstance-smbbn --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-hscaleoninstance-smbbn patched `kbcli cluster delete-ops --name mongodb-rxyipy-hscaleoninstance-smbbn --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-hscaleoninstance-smbbn deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fe3c8fd6bbc1f35e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T11:17:30.245+00:00: You are running this process as the root user, which is not recommended 2025-06-19T11:17:30.245+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fe53ca4eb1004a5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T12:07:51.355+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:07:51.355+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop mongodb-rxyipy --auto-approve --force=true --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-stop-25rdw created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-stop-25rdw -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-stop-25rdw ns-azgwq Stop mongodb-rxyipy mongodb Running 0/3 Jun 19,2025 20:11 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Stopping Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-stop-25rdw ns-azgwq Stop mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 20:11 UTC+0800 check ops status done ops_status:mongodb-rxyipy-stop-25rdw ns-azgwq Stop mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 20:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-stop-25rdw --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-stop-25rdw patched `kbcli cluster delete-ops --name mongodb-rxyipy-stop-25rdw --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-stop-25rdw deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start mongodb-rxyipy --force=true --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-start-jvjgs created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-start-jvjgs -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-start-jvjgs ns-azgwq Start mongodb-rxyipy mongodb Running 0/3 Jun 19,2025 20:12 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 20:12 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 20:12 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 20:13 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-0;secondary: mongodb-rxyipy-mongodb-1 mongodb-rxyipy-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-start-jvjgs ns-azgwq Start mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 20:12 UTC+0800 check ops status done ops_status:mongodb-rxyipy-start-jvjgs ns-azgwq Start mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 20:12 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-start-jvjgs --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-start-jvjgs patched `kbcli cluster delete-ops --name mongodb-rxyipy-start-jvjgs --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-start-jvjgs deleted `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-0: out of mongodb-rxyipy-mongodb-0(primary), mongodb-rxyipy-mongodb-1(secondary), mongodb-rxyipy-mongodb-2(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ff0a7fb958b6e75e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-0.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:12:48.162+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:12:48.162+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-1 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853ff2104394b555e5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T12:13:28.605+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:13:28.606+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster configure component_tmp: mongodb check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure mongodb-rxyipy --auto-approve --force=true --set systemLog.quiet=false,systemLog.verbosity=1 --components mongodb --config-spec mongodb-config --config-file mongodb.conf --namespace ns-azgwq ` Will updated configure file meta: ConfigSpec: mongodb-config ConfigFile: mongodb.conf ComponentName: mongodb ClusterName: mongodb-rxyipy OpsRequest mongodb-rxyipy-reconfiguring-rkgrs created successfully, you can view the progress: kbcli cluster describe-ops mongodb-rxyipy-reconfiguring-rkgrs -n ns-azgwq check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-reconfiguring-rkgrs ns-azgwq Reconfiguring mongodb-rxyipy mongodb Running -/- Jun 19,2025 20:14 UTC+0800 check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq Delete Updating Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 20:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 20:15 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 20:15 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME mongodb-rxyipy-reconfiguring-rkgrs ns-azgwq Reconfiguring mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 20:14 UTC+0800 check ops status done ops_status:mongodb-rxyipy-reconfiguring-rkgrs ns-azgwq Reconfiguring mongodb-rxyipy mongodb Succeed 3/3 Jun 19,2025 20:14 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests mongodb-rxyipy-reconfiguring-rkgrs --namespace ns-azgwq ` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-reconfiguring-rkgrs patched `kbcli cluster delete-ops --name mongodb-rxyipy-reconfiguring-rkgrs --force --auto-approve --namespace ns-azgwq ` OpsRequest mongodb-rxyipy-reconfiguring-rkgrs deleted check config variables Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: false configure:[systemLog.quiet] result actual:[false] equal expected:[false] Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 1 configure:[systemLog.verbosity] result actual:[1] equal expected:[1] `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6853fff3d24c8dba965e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:15:31.117+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:15:31.117+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 6854000be59e2fe9e35e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ To help improve our products, anonymous usage data is collected and sent to MongoDB periodically (https://www.mongodb.com/legal/privacy-policy). You can opt-out by running the disableTelemetry() command. ------ The server generated these startup warnings when booting 2025-06-19T12:16:45.849+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:16:45.849+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update mongodb-rxyipy --termination-policy=WipeOut --namespace ns-azgwq ` cluster.apps.kubeblocks.io/mongodb-rxyipy updated check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 20:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 20:15 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 20:15 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done cluster volume-snapshot backup `kbcli cluster backup mongodb-rxyipy --method volume-snapshot --namespace ns-azgwq ` Backup backup-ns-azgwq-mongodb-rxyipy-20250619201916 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-azgwq-mongodb-rxyipy-20250619201916 -n ns-azgwq check backup status `kbcli cluster list-backups mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-azgwq-mongodb-rxyipy-20250619201916 ns-azgwq mongodb-rxyipy volume-snapshot Running Jun 19,2025 20:19 UTC+0800 backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running backup_status:mongodb-rxyipy-volume-snapshot-Running check backup status done backup_status:backup-ns-azgwq-mongodb-rxyipy-20250619201916 ns-azgwq mongodb-rxyipy volume-snapshot Completed 5Gi 69s Jun 19,2025 20:19 UTC+0800 Jun 19,2025 20:20 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "mongodb-rxyipy-backup" not found `kbcli cluster describe-backup backup-ns-azgwq-mongodb-rxyipy-20250619201916 --namespace ns-azgwq ` Name: backup-ns-azgwq-mongodb-rxyipy-20250619201916 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: volume-snapshot Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 5Gi ActionSet Name: mongodb-volumesnapshot Duration: 69s Start Time: Jun 19,2025 20:19 UTC+0800 Completion Time: Jun 19,2025 20:20 UTC+0800 Time Range Start: Jun 19,2025 20:19 UTC+0800 Time Range End: Jun 19,2025 20:19 UTC+0800 Warning Events: `kbcli cluster restore mongodb-rxyipy-backup --backup backup-ns-azgwq-mongodb-rxyipy-20250619201916 --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup created check cluster status `kbcli cluster list mongodb-rxyipy-backup --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy-backup ns-azgwq WipeOut Jun 19,2025 20:20 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy-backup --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-backup-mongodb-0 ns-azgwq mongodb-rxyipy-backup mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 20:21 UTC+0800 mongodb-rxyipy-backup-mongodb-1 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 20:21 UTC+0800 mongodb-rxyipy-backup-mongodb-2 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 20:21 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-backup-mongodb-0;secondary: mongodb-rxyipy-backup-mongodb-1 mongodb-rxyipy-backup-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq` check cluster connect done `kbcli cluster describe-backup backup-ns-azgwq-mongodb-rxyipy-20250619201916 --namespace ns-azgwq ` Name: backup-ns-azgwq-mongodb-rxyipy-20250619201916 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: volume-snapshot Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 5Gi ActionSet Name: mongodb-volumesnapshot Duration: 69s Start Time: Jun 19,2025 20:19 UTC+0800 Completion Time: Jun 19,2025 20:20 UTC+0800 Time Range Start: Jun 19,2025 20:19 UTC+0800 Time Range End: Jun 19,2025 20:19 UTC+0800 Warning Events: cluster connect `echo 'rs.status()' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance mongodb-rxyipy-backup-mongodb-0: out of mongodb-rxyipy-backup-mongodb-0(primary), mongodb-rxyipy-backup-mongodb-1(secondary), mongodb-rxyipy-backup-mongodb-2(secondary) Current Mongosh Log ID: 685401332ff3f4a2ae5e739b Connecting to: mongodb://@mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-backup-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:21:50.822+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:21:50.822+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-backup-mongodb [primary] admin> *** set: 'mongodb-rxyipy-backup-mongodb', date: ISODate('2025-06-19T12:23:23.621Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-06-19T12:23:23.015Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-06-19T12:23:23.015Z'), lastDurableWallTime: ISODate('2025-06-19T12:23:23.015Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1750335752, i: 1 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-06-19T12:22:42.989Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1750335752, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1750335752, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 2, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-06-19T12:22:42.998Z'), wMajorityWriteAvailabilityDate: ISODate('2025-06-19T12:22:43.811Z') ***, members: [ *** _id: 0, name: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 100, optime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:23:23.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:23:23.015Z'), lastDurableWallTime: ISODate('2025-06-19T12:23:23.015Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp(*** t: 1750335762, i: 1 ***), electionDate: ISODate('2025-06-19T12:22:42.000Z'), configVersion: 1, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-rxyipy-backup-mongodb-1.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 51, optime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:23:23.000Z'), optimeDurableDate: ISODate('2025-06-19T12:23:23.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:23:23.015Z'), lastDurableWallTime: ISODate('2025-06-19T12:23:23.015Z'), lastHeartbeat: ISODate('2025-06-19T12:23:23.411Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:23:22.312Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 ***, *** _id: 2, name: 'mongodb-rxyipy-backup-mongodb-2.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 51, optime: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750335803, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:23:23.000Z'), optimeDurableDate: ISODate('2025-06-19T12:23:23.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:23:23.015Z'), lastDurableWallTime: ISODate('2025-06-19T12:23:23.015Z'), lastHeartbeat: ISODate('2025-06-19T12:23:23.413Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:23:22.411Z'), pingMs: Long('15'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1750335803, i: 1 ***), signature: *** hash: Binary.createFromBase64('VR4FwdSGboPc3WRQOFId/gkr/ks=', 0), keyId: Long('7517602608194781190') *** ***, operationTime: Timestamp(*** t: 1750335803, i: 1 ***) *** mongodb-rxyipy-backup-mongodb [primary] admin> connect cluster Success delete cluster mongodb-rxyipy-backup `kbcli cluster delete mongodb-rxyipy-backup --auto-approve --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup deleted pod_info:mongodb-rxyipy-backup-mongodb-0 3/3 Terminating 0 2m25s mongodb-rxyipy-backup-mongodb-1 3/3 Terminating 0 2m7s mongodb-rxyipy-backup-mongodb-2 3/3 Terminating 0 108s No resources found in ns-azgwq namespace. delete cluster pod done No resources found in ns-azgwq namespace. check cluster resource non-exist OK: pvc No resources found in ns-azgwq namespace. delete cluster done No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-azgwq-mongodb-rxyipy-20250619201916 --namespace ns-azgwq ` backup.dataprotection.kubeblocks.io/backup-ns-azgwq-mongodb-rxyipy-20250619201916 patched `kbcli cluster delete-backup mongodb-rxyipy --name backup-ns-azgwq-mongodb-rxyipy-20250619201916 --force --auto-approve --namespace ns-azgwq ` Backup backup-ns-azgwq-mongodb-rxyipy-20250619201916 deleted No opsrequests found in ns-azgwq namespace. cluster datafile backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mongodb-rxyipy --method datafile --namespace ns-azgwq ` Backup backup-ns-azgwq-mongodb-rxyipy-20250619202412 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-azgwq-mongodb-rxyipy-20250619202412 -n ns-azgwq check backup status `kbcli cluster list-backups mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-azgwq-mongodb-rxyipy-20250619202412 ns-azgwq mongodb-rxyipy datafile Running Jun 19,2025 20:24 UTC+0800 backup_status:mongodb-rxyipy-datafile-Running backup_status:mongodb-rxyipy-datafile-Running backup_status:mongodb-rxyipy-datafile-Running backup_status:mongodb-rxyipy-datafile-Running backup_status:mongodb-rxyipy-datafile-Running check backup status done backup_status:backup-ns-azgwq-mongodb-rxyipy-20250619202412 ns-azgwq mongodb-rxyipy datafile Completed 603251 25s Jun 19,2025 20:24 UTC+0800 Jun 19,2025 20:24 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "mongodb-rxyipy-backup" not found `kbcli cluster describe-backup backup-ns-azgwq-mongodb-rxyipy-20250619202412 --namespace ns-azgwq ` Name: backup-ns-azgwq-mongodb-rxyipy-20250619202412 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: datafile Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 603251 ActionSet Name: mongodb-physical-backup Repository: backuprepo-kbcli-test Duration: 25s Start Time: Jun 19,2025 20:24 UTC+0800 Completion Time: Jun 19,2025 20:24 UTC+0800 Path: /ns-azgwq/mongodb-rxyipy-9c253df5-5b4a-48d1-87c0-8e9ca6fdaf97/mongodb/backup-ns-azgwq-mongodb-rxyipy-20250619202412 Time Range Start: Jun 19,2025 20:24 UTC+0800 Time Range End: Jun 19,2025 20:24 UTC+0800 Warning Events: `kbcli cluster restore mongodb-rxyipy-backup --backup backup-ns-azgwq-mongodb-rxyipy-20250619202412 --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup created check cluster status `kbcli cluster list mongodb-rxyipy-backup --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy-backup ns-azgwq WipeOut Jun 19,2025 20:24 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy-backup --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-backup-mongodb-0 ns-azgwq mongodb-rxyipy-backup mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 20:25 UTC+0800 mongodb-rxyipy-backup-mongodb-1 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 20:25 UTC+0800 mongodb-rxyipy-backup-mongodb-2 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 20:26 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-backup-mongodb-0;secondary: mongodb-rxyipy-backup-mongodb-1 mongodb-rxyipy-backup-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq` check cluster connect done `kbcli cluster describe-backup backup-ns-azgwq-mongodb-rxyipy-20250619202412 --namespace ns-azgwq ` Name: backup-ns-azgwq-mongodb-rxyipy-20250619202412 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: datafile Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 603251 ActionSet Name: mongodb-physical-backup Repository: backuprepo-kbcli-test Duration: 25s Start Time: Jun 19,2025 20:24 UTC+0800 Completion Time: Jun 19,2025 20:24 UTC+0800 Path: /ns-azgwq/mongodb-rxyipy-9c253df5-5b4a-48d1-87c0-8e9ca6fdaf97/mongodb/backup-ns-azgwq-mongodb-rxyipy-20250619202412 Time Range Start: Jun 19,2025 20:24 UTC+0800 Time Range End: Jun 19,2025 20:24 UTC+0800 Warning Events: cluster connect `echo 'rs.status()' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance mongodb-rxyipy-backup-mongodb-0: out of mongodb-rxyipy-backup-mongodb-0(primary), mongodb-rxyipy-backup-mongodb-1(secondary), mongodb-rxyipy-backup-mongodb-2(secondary) Current Mongosh Log ID: 685402436774ded0105e739b Connecting to: mongodb://@mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-backup-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:26:15.035+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:26:15.035+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-backup-mongodb [primary] admin> *** set: 'mongodb-rxyipy-backup-mongodb', date: ISODate('2025-06-19T12:27:56.316Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-06-19T12:27:53.114Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-06-19T12:27:53.114Z'), lastDurableWallTime: ISODate('2025-06-19T12:27:53.114Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1750336021, i: 1 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-06-19T12:27:13.026Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1750336021, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1750336021, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 2, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-06-19T12:27:13.036Z'), wMajorityWriteAvailabilityDate: ISODate('2025-06-19T12:27:13.518Z') ***, members: [ *** _id: 0, name: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 109, optime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:27:53.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:27:53.114Z'), lastDurableWallTime: ISODate('2025-06-19T12:27:53.114Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp(*** t: 1750336033, i: 1 ***), electionDate: ISODate('2025-06-19T12:27:13.000Z'), configVersion: 1, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-rxyipy-backup-mongodb-1.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 54, optime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:27:53.000Z'), optimeDurableDate: ISODate('2025-06-19T12:27:53.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:27:53.114Z'), lastDurableWallTime: ISODate('2025-06-19T12:27:53.114Z'), lastHeartbeat: ISODate('2025-06-19T12:27:55.615Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:27:55.814Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 ***, *** _id: 2, name: 'mongodb-rxyipy-backup-mongodb-2.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 54, optime: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750336073, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:27:53.000Z'), optimeDurableDate: ISODate('2025-06-19T12:27:53.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:27:53.114Z'), lastDurableWallTime: ISODate('2025-06-19T12:27:53.114Z'), lastHeartbeat: ISODate('2025-06-19T12:27:55.614Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:27:55.814Z'), pingMs: Long('19'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1750336073, i: 1 ***), signature: *** hash: Binary.createFromBase64('ObhIFczEFZfBs+BwuzhirqCYge0=', 0), keyId: Long('7517602608194781190') *** ***, operationTime: Timestamp(*** t: 1750336073, i: 1 ***) *** mongodb-rxyipy-backup-mongodb [primary] admin> connect cluster Success delete cluster mongodb-rxyipy-backup `kbcli cluster delete mongodb-rxyipy-backup --auto-approve --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup deleted pod_info:mongodb-rxyipy-backup-mongodb-0 3/3 Terminating 0 2m37s mongodb-rxyipy-backup-mongodb-1 3/3 Terminating 0 2m16s mongodb-rxyipy-backup-mongodb-2 3/3 Terminating 0 115s No resources found in ns-azgwq namespace. delete cluster pod done No resources found in ns-azgwq namespace. check cluster resource non-exist OK: pvc No resources found in ns-azgwq namespace. delete cluster done No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. cluster rebulid instances apiVersion: apps.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: mongodb-rxyipy-rebuildinstance- namespace: ns-azgwq spec: type: RebuildInstance clusterName: mongodb-rxyipy force: true rebuildFrom: - componentName: mongodb instances: - name: mongodb-rxyipy-mongodb-0 inPlace: false check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_mongodb-rxyipy.yaml` opsrequest.apps.kubeblocks.io/mongodb-rxyipy-rebuildinstance-tpq6l created create test_ops_cluster_mongodb-rxyipy.yaml Success `rm -rf test_ops_cluster_mongodb-rxyipy.yaml` check ops status `kbcli cluster list-ops mongodb-rxyipy --status all --namespace ns-azgwq ` No opsrequests found in ns-azgwq namespace. No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: No opsrequests found in ns-azgwq namespace. ops_status: not found ops rebuildinstance check cluster status `kbcli cluster list mongodb-rxyipy --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy ns-azgwq WipeOut Running Jun 19,2025 18:16 UTC+0800 app.kubernetes.io/instance=mongodb-rxyipy check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-mongodb-0 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-181.us-west-2.compute.internal/172.31.2.181 Jun 19,2025 20:16 UTC+0800 mongodb-rxyipy-mongodb-1 ns-azgwq mongodb-rxyipy mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-6-232.us-west-2.compute.internal/172.31.6.232 Jun 19,2025 20:15 UTC+0800 mongodb-rxyipy-mongodb-2 ns-azgwq mongodb-rxyipy mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-10-206.us-west-2.compute.internal/172.31.10.206 Jun 19,2025 20:15 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-mongodb-2;secondary: mongodb-rxyipy-mongodb-0 mongodb-rxyipy-mongodb-1 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq` check cluster connect done `echo 'db.col.find()' | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check data: Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 68540371fa08720dfd5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:15:31.117+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:15:31.117+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [primary] admin> check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=mongodb-rxyipy` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.username***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.password***"` `kubectl get secrets mongodb-rxyipy-mongodb-account-root -o jsonpath="***.data.port***"` DB_USERNAME:root;DB_PASSWORD:0M3A04b2z7kh69ij;DB_PORT:27017;DB_DATABASE: No resources found in ns-azgwq namespace. `echo "echo \"db.col.find().readPref('secondary')\" | mongosh --host mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local --port 27017 -u root -p 0M3A04b2z7kh69ij --authenticationDatabase admin admin " | kubectl exec -it mongodb-rxyipy-mongodb-0 --namespace ns-azgwq -- bash ` check readonly data: Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 685403896e3cb2c1055e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-mongodb-ro.ns-azgwq.svc.cluster.local:27017/admin?directConnection=true&authSource=admin&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:16:12.397+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:16:12.397+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [direct: secondary] admin> [ *** _id: ObjectId('6853e54063ea60a2f55e739c'), a: 'egbjb' *** ] mongodb-rxyipy-mongodb [direct: secondary] admin> check cluster readonly data consistent Success check db_client batch data count `echo "db.executions_loop_table.estimatedDocumentCount();" | kbcli cluster connect mongodb-rxyipy --namespace ns-azgwq ` check db_client batch data Success cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-azgwq-mongodb-rxyipy-20250619202412 --namespace ns-azgwq ` backup.dataprotection.kubeblocks.io/backup-ns-azgwq-mongodb-rxyipy-20250619202412 patched `kbcli cluster delete-backup mongodb-rxyipy --name backup-ns-azgwq-mongodb-rxyipy-20250619202412 --force --auto-approve --namespace ns-azgwq ` Backup backup-ns-azgwq-mongodb-rxyipy-20250619202412 deleted No opsrequests found in ns-azgwq namespace. `kubectl get backupschedule -l app.kubernetes.io/instance=mongodb-rxyipy ` `kubectl get backupschedule mongodb-rxyipy-mongodb-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched check backup status `kbcli cluster list-backups mongodb-rxyipy --namespace ns-azgwq ` No backups found in ns-azgwq namespace. No backups found in ns-azgwq namespace. backup_status:-- No backups found in ns-azgwq namespace. backup_status:-- `kubectl get backupschedule -l app.kubernetes.io/instance=mongodb-rxyipy ` `kubectl get backupschedule mongodb-rxyipy-mongodb-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backup_status:mongodb-rxyipy-datafile-Running check backup status done backup_status:mongodb-rxyipy-datafile-20250619123400 ns-azgwq mongodb-rxyipy datafile Completed 715952 10s Jun 19,2025 20:34 UTC+0800 Jun 19,2025 20:34 UTC+0800 Jun 26,2025 20:34 UTC+0800 `kubectl get backupschedule -l app.kubernetes.io/instance=mongodb-rxyipy ` `kubectl get backupschedule mongodb-rxyipy-mongodb-backup-schedule -ojsonpath='***.spec.schedules[*].backupMethod***' ` backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) backupschedule.dataprotection.kubeblocks.io/mongodb-rxyipy-mongodb-backup-schedule patched (no change) cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "mongodb-rxyipy-backup" not found `kbcli cluster describe-backup mongodb-rxyipy-datafile-20250619123400 --namespace ns-azgwq ` Name: mongodb-rxyipy-datafile-20250619123400 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: datafile Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 715952 ActionSet Name: mongodb-physical-backup Repository: backuprepo-kbcli-test Duration: 10s Expiration Time: Jun 26,2025 20:34 UTC+0800 Start Time: Jun 19,2025 20:34 UTC+0800 Completion Time: Jun 19,2025 20:34 UTC+0800 Path: /ns-azgwq/mongodb-rxyipy-9c253df5-5b4a-48d1-87c0-8e9ca6fdaf97/mongodb/mongodb-rxyipy-datafile-20250619123400 Time Range Start: Jun 19,2025 20:33 UTC+0800 Time Range End: Jun 19,2025 20:34 UTC+0800 Warning Events: `kbcli cluster restore mongodb-rxyipy-backup --backup mongodb-rxyipy-datafile-20250619123400 --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup created check cluster status `kbcli cluster list mongodb-rxyipy-backup --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy-backup ns-azgwq WipeOut Jun 19,2025 20:34 UTC+0800 cluster_status: cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy-backup --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-backup-mongodb-0 ns-azgwq mongodb-rxyipy-backup mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 20:34 UTC+0800 mongodb-rxyipy-backup-mongodb-1 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 20:35 UTC+0800 mongodb-rxyipy-backup-mongodb-2 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 20:35 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-backup-mongodb-0;secondary: mongodb-rxyipy-backup-mongodb-1 mongodb-rxyipy-backup-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq` check cluster connect done `kbcli cluster describe-backup mongodb-rxyipy-datafile-20250619123400 --namespace ns-azgwq ` Name: mongodb-rxyipy-datafile-20250619123400 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: datafile Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 715952 ActionSet Name: mongodb-physical-backup Repository: backuprepo-kbcli-test Duration: 10s Expiration Time: Jun 26,2025 20:34 UTC+0800 Start Time: Jun 19,2025 20:34 UTC+0800 Completion Time: Jun 19,2025 20:34 UTC+0800 Path: /ns-azgwq/mongodb-rxyipy-9c253df5-5b4a-48d1-87c0-8e9ca6fdaf97/mongodb/mongodb-rxyipy-datafile-20250619123400 Time Range Start: Jun 19,2025 20:33 UTC+0800 Time Range End: Jun 19,2025 20:34 UTC+0800 Warning Events: cluster connect `echo 'rs.status()' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance mongodb-rxyipy-backup-mongodb-0: out of mongodb-rxyipy-backup-mongodb-0(primary), mongodb-rxyipy-backup-mongodb-1(secondary), mongodb-rxyipy-backup-mongodb-2(secondary) Current Mongosh Log ID: 6854047af46bed058f5e739b Connecting to: mongodb://@mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-backup-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:35:38.433+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:35:38.433+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-backup-mongodb [primary] admin> *** set: 'mongodb-rxyipy-backup-mongodb', date: ISODate('2025-06-19T12:37:22.527Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-06-19T12:37:17.714Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-06-19T12:37:17.714Z'), lastDurableWallTime: ISODate('2025-06-19T12:37:17.714Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1750336586, i: 1 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-06-19T12:36:37.623Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1750336586, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1750336586, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 2, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), numCatchUpOps: Long('0'), newTermStartDate: ISODate('2025-06-19T12:36:37.633Z'), wMajorityWriteAvailabilityDate: ISODate('2025-06-19T12:36:37.927Z') ***, members: [ *** _id: 0, name: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 111, optime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:37:17.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:37:17.714Z'), lastDurableWallTime: ISODate('2025-06-19T12:37:17.714Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: '', electionTime: Timestamp(*** t: 1750336597, i: 1 ***), electionDate: ISODate('2025-06-19T12:36:37.000Z'), configVersion: 1, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-rxyipy-backup-mongodb-1.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 56, optime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:37:17.000Z'), optimeDurableDate: ISODate('2025-06-19T12:37:17.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:37:17.714Z'), lastDurableWallTime: ISODate('2025-06-19T12:37:17.714Z'), lastHeartbeat: ISODate('2025-06-19T12:37:22.212Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:37:22.416Z'), pingMs: Long('22'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 ***, *** _id: 2, name: 'mongodb-rxyipy-backup-mongodb-2.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 56, optime: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750336637, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:37:17.000Z'), optimeDurableDate: ISODate('2025-06-19T12:37:17.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:37:17.714Z'), lastDurableWallTime: ISODate('2025-06-19T12:37:17.714Z'), lastHeartbeat: ISODate('2025-06-19T12:37:22.212Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:37:22.311Z'), pingMs: Long('19'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 1, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1750336637, i: 1 ***), signature: *** hash: Binary.createFromBase64('+d/U38GX2WI/jPnG8o0mvx3dXZo=', 0), keyId: Long('7517602608194781190') *** ***, operationTime: Timestamp(*** t: 1750336637, i: 1 ***) *** mongodb-rxyipy-backup-mongodb [primary] admin> connect cluster Success delete cluster mongodb-rxyipy-backup `kbcli cluster delete mongodb-rxyipy-backup --auto-approve --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup deleted pod_info:mongodb-rxyipy-backup-mongodb-0 3/3 Terminating 0 2m35s mongodb-rxyipy-backup-mongodb-1 3/3 Terminating 0 2m18s mongodb-rxyipy-backup-mongodb-2 3/3 Terminating 0 2m1s No resources found in ns-azgwq namespace. delete cluster pod done No resources found in ns-azgwq namespace. check cluster resource non-exist OK: pvc No resources found in ns-azgwq namespace. delete cluster done No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups mongodb-rxyipy-datafile-20250619123400 --namespace ns-azgwq ` backup.dataprotection.kubeblocks.io/mongodb-rxyipy-datafile-20250619123400 patched `kbcli cluster delete-backup mongodb-rxyipy --name mongodb-rxyipy-datafile-20250619123400 --force --auto-approve --namespace ns-azgwq ` Backup mongodb-rxyipy-datafile-20250619123400 deleted No opsrequests found in ns-azgwq namespace. `db.msg.drop();db.createCollection("msg");db.msg.insertOne(***msg: "kbcli-test-data-rxyipy0",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy1",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy2",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy3",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy4",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy5",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy6",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy7",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy8",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy9",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy10",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy11",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy12",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy13",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy14",time: new Date()***);db.msg.insertOne(***msg: "kbcli-test-data-rxyipy15",time: new Date()***);` Connect to instance mongodb-rxyipy-mongodb-2: out of mongodb-rxyipy-mongodb-2(primary), mongodb-rxyipy-mongodb-0(secondary), mongodb-rxyipy-mongodb-1(secondary) Unable to use a TTY - input is not a terminal or the right kind of file Current Mongosh Log ID: 685404a9547a9251fe5e739b Connecting to: mongodb://@mongodb-rxyipy-mongodb-2.mongodb-rxyipy-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:15:31.117+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:15:31.117+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-mongodb [primary] admin> *** acknowledged: true, insertedId: ObjectId('685404bb547a9251fe5e73ab') *** mongodb-rxyipy-mongodb [primary] admin> cluster dump backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-w7bzj -n kb-mqtcr -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-mqtcr get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-mqtcr -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-mqtcr.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-5c88d6b598-9v8l5 --namespace kb-mqtcr -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup mongodb-rxyipy --method dump --namespace ns-azgwq ` Backup backup-ns-azgwq-mongodb-rxyipy-20250619203834 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-azgwq-mongodb-rxyipy-20250619203834 -n ns-azgwq check backup status `kbcli cluster list-backups mongodb-rxyipy --namespace ns-azgwq ` NAME NAMESPACE SOURCE-CLUSTER METHOD STATUS TOTAL-SIZE DURATION CREATE-TIME COMPLETION-TIME EXPIRATION backup-ns-azgwq-mongodb-rxyipy-20250619203834 ns-azgwq mongodb-rxyipy dump Running Jun 19,2025 20:38 UTC+0800 backup_status:mongodb-rxyipy-dump-Running backup_status:mongodb-rxyipy-dump-Running check backup status done backup_status:backup-ns-azgwq-mongodb-rxyipy-20250619203834 ns-azgwq mongodb-rxyipy dump Completed 2890 11s Jun 19,2025 20:38 UTC+0800 Jun 19,2025 20:38 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "mongodb-rxyipy-backup" not found `kbcli cluster describe-backup backup-ns-azgwq-mongodb-rxyipy-20250619203834 --namespace ns-azgwq ` Name: backup-ns-azgwq-mongodb-rxyipy-20250619203834 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: dump Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 2890 ActionSet Name: mongodb-dump Repository: backuprepo-kbcli-test Duration: 11s Start Time: Jun 19,2025 20:38 UTC+0800 Completion Time: Jun 19,2025 20:38 UTC+0800 Path: /ns-azgwq/mongodb-rxyipy-9c253df5-5b4a-48d1-87c0-8e9ca6fdaf97/mongodb/backup-ns-azgwq-mongodb-rxyipy-20250619203834 Time Range Start: Jun 19,2025 20:38 UTC+0800 Time Range End: Jun 19,2025 20:38 UTC+0800 Warning Events: `kbcli cluster restore mongodb-rxyipy-backup --backup backup-ns-azgwq-mongodb-rxyipy-20250619203834 --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup created check cluster status `kbcli cluster list mongodb-rxyipy-backup --show-labels --namespace ns-azgwq ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS mongodb-rxyipy-backup ns-azgwq WipeOut Jun 19,2025 20:38 UTC+0800 cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances mongodb-rxyipy-backup --namespace ns-azgwq ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME mongodb-rxyipy-backup-mongodb-0 ns-azgwq mongodb-rxyipy-backup mongodb Running primary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-2-66.us-west-2.compute.internal/172.31.2.66 Jun 19,2025 20:38 UTC+0800 mongodb-rxyipy-backup-mongodb-1 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-12-104.us-west-2.compute.internal/172.31.12.104 Jun 19,2025 20:39 UTC+0800 mongodb-rxyipy-backup-mongodb-2 ns-azgwq mongodb-rxyipy-backup mongodb Running secondary us-west-2a 200m / 200m 644245094400m / 644245094400m data:5Gi ip-172-31-1-34.us-west-2.compute.internal/172.31.1.34 Jun 19,2025 20:39 UTC+0800 check pod status done check cluster role check cluster role done primary: mongodb-rxyipy-backup-mongodb-0;secondary: mongodb-rxyipy-backup-mongodb-1 mongodb-rxyipy-backup-mongodb-2 check cluster connect `echo '' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq` check cluster connect done `kbcli cluster describe-backup backup-ns-azgwq-mongodb-rxyipy-20250619203834 --namespace ns-azgwq ` Name: backup-ns-azgwq-mongodb-rxyipy-20250619203834 Cluster: mongodb-rxyipy Namespace: ns-azgwq Spec: Method: dump Policy Name: mongodb-rxyipy-mongodb-backup-policy Status: Phase: Completed Total Size: 2890 ActionSet Name: mongodb-dump Repository: backuprepo-kbcli-test Duration: 11s Start Time: Jun 19,2025 20:38 UTC+0800 Completion Time: Jun 19,2025 20:38 UTC+0800 Path: /ns-azgwq/mongodb-rxyipy-9c253df5-5b4a-48d1-87c0-8e9ca6fdaf97/mongodb/backup-ns-azgwq-mongodb-rxyipy-20250619203834 Time Range Start: Jun 19,2025 20:38 UTC+0800 Time Range End: Jun 19,2025 20:38 UTC+0800 Warning Events: `db.msg.find();` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance mongodb-rxyipy-backup-mongodb-0: out of mongodb-rxyipy-backup-mongodb-0(primary), mongodb-rxyipy-backup-mongodb-1(secondary), mongodb-rxyipy-backup-mongodb-2(secondary) Current Mongosh Log ID: 68540541f76cd1929b5e739b Connecting to: mongodb://@mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-backup-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:39:16.259+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:39:16.260+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-backup-mongodb [primary] admin> [ *** _id: ObjectId('685404b8547a9251fe5e739c'), msg: 'kbcli-test-data-rxyipy0', time: ISODate('2025-06-19T12:38:16.008Z') ***, *** _id: ObjectId('685404b8547a9251fe5e739d'), msg: 'kbcli-test-data-rxyipy1', time: ISODate('2025-06-19T12:38:16.603Z') ***, *** _id: ObjectId('685404b8547a9251fe5e739e'), msg: 'kbcli-test-data-rxyipy2', time: ISODate('2025-06-19T12:38:16.807Z') ***, *** _id: ObjectId('685404b9547a9251fe5e739f'), msg: 'kbcli-test-data-rxyipy3', time: ISODate('2025-06-19T12:38:17.000Z') ***, *** _id: ObjectId('685404b9547a9251fe5e73a0'), msg: 'kbcli-test-data-rxyipy4', time: ISODate('2025-06-19T12:38:17.298Z') ***, *** _id: ObjectId('685404b9547a9251fe5e73a1'), msg: 'kbcli-test-data-rxyipy5', time: ISODate('2025-06-19T12:38:17.505Z') ***, *** _id: ObjectId('685404b9547a9251fe5e73a2'), msg: 'kbcli-test-data-rxyipy6', time: ISODate('2025-06-19T12:38:17.699Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a3'), msg: 'kbcli-test-data-rxyipy7', time: ISODate('2025-06-19T12:38:17.999Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a4'), msg: 'kbcli-test-data-rxyipy8', time: ISODate('2025-06-19T12:38:18.299Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a5'), msg: 'kbcli-test-data-rxyipy9', time: ISODate('2025-06-19T12:38:18.407Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a6'), msg: 'kbcli-test-data-rxyipy10', time: ISODate('2025-06-19T12:38:18.504Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a7'), msg: 'kbcli-test-data-rxyipy11', time: ISODate('2025-06-19T12:38:18.602Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a8'), msg: 'kbcli-test-data-rxyipy12', time: ISODate('2025-06-19T12:38:18.798Z') ***, *** _id: ObjectId('685404ba547a9251fe5e73a9'), msg: 'kbcli-test-data-rxyipy13', time: ISODate('2025-06-19T12:38:18.901Z') ***, *** _id: ObjectId('685404bb547a9251fe5e73aa'), msg: 'kbcli-test-data-rxyipy14', time: ISODate('2025-06-19T12:38:19.100Z') ***, *** _id: ObjectId('685404bb547a9251fe5e73ab'), msg: 'kbcli-test-data-rxyipy15', time: ISODate('2025-06-19T12:38:19.208Z') *** ] mongodb-rxyipy-backup-mongodb [primary] admin> dump backup check data Success cluster connect `echo 'rs.status()' | kbcli cluster connect mongodb-rxyipy-backup --namespace ns-azgwq ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance mongodb-rxyipy-backup-mongodb-0: out of mongodb-rxyipy-backup-mongodb-0(primary), mongodb-rxyipy-backup-mongodb-1(secondary), mongodb-rxyipy-backup-mongodb-2(secondary) Current Mongosh Log ID: 685405520b256cef9f5e739b Connecting to: mongodb://@mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017/admin?replicaSet=mongodb-rxyipy-backup-mongodb&appName=mongosh+2.3.0 Using MongoDB: 6.0.16 Using Mongosh: 2.3.0 For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ ------ The server generated these startup warnings when booting 2025-06-19T12:39:16.259+00:00: You are running this process as the root user, which is not recommended 2025-06-19T12:39:16.260+00:00: vm.max_map_count is too low ------ mongodb-rxyipy-backup-mongodb [primary] admin> *** set: 'mongodb-rxyipy-backup-mongodb', date: ISODate('2025-06-19T12:40:59.526Z'), myState: 1, term: Long('1'), syncSourceHost: '', syncSourceId: -1, heartbeatIntervalMillis: Long('2000'), majorityVoteCount: 2, writeMajorityCount: 2, votingMembersCount: 3, writableVotingMembersCount: 3, optimes: *** lastCommittedOpTime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, lastCommittedWallTime: ISODate('2025-06-19T12:40:57.228Z'), readConcernMajorityOpTime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, appliedOpTime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, durableOpTime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, lastAppliedWallTime: ISODate('2025-06-19T12:40:57.228Z'), lastDurableWallTime: ISODate('2025-06-19T12:40:57.228Z') ***, lastStableRecoveryTimestamp: Timestamp(*** t: 1750336806, i: 6 ***), electionCandidateMetrics: *** lastElectionReason: 'electionTimeout', lastElectionDate: ISODate('2025-06-19T12:39:16.927Z'), electionTerm: Long('1'), lastCommittedOpTimeAtElection: *** ts: Timestamp(*** t: 1750336756, i: 1 ***), t: Long('-1') ***, lastSeenOpTimeAtElection: *** ts: Timestamp(*** t: 1750336756, i: 1 ***), t: Long('-1') ***, numVotesNeeded: 1, priorityAtElection: 2, electionTimeoutMillis: Long('10000'), newTermStartDate: ISODate('2025-06-19T12:39:17.044Z'), wMajorityWriteAvailabilityDate: ISODate('2025-06-19T12:39:17.148Z') ***, members: [ *** _id: 0, name: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 1, stateStr: 'PRIMARY', uptime: 109, optime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:40:57.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:40:57.228Z'), lastDurableWallTime: ISODate('2025-06-19T12:40:57.228Z'), syncSourceHost: '', syncSourceId: -1, infoMessage: 'Could not find member to sync from', electionTime: Timestamp(*** t: 1750336756, i: 2 ***), electionDate: ISODate('2025-06-19T12:39:16.000Z'), configVersion: 5, configTerm: 1, self: true, lastHeartbeatMessage: '' ***, *** _id: 1, name: 'mongodb-rxyipy-backup-mongodb-1.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 77, optime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:40:57.000Z'), optimeDurableDate: ISODate('2025-06-19T12:40:57.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:40:57.228Z'), lastDurableWallTime: ISODate('2025-06-19T12:40:57.228Z'), lastHeartbeat: ISODate('2025-06-19T12:40:57.627Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:40:59.331Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 5, configTerm: 1 ***, *** _id: 2, name: 'mongodb-rxyipy-backup-mongodb-2.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', health: 1, state: 2, stateStr: 'SECONDARY', uptime: 60, optime: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, optimeDurable: *** ts: Timestamp(*** t: 1750336857, i: 1 ***), t: Long('1') ***, optimeDate: ISODate('2025-06-19T12:40:57.000Z'), optimeDurableDate: ISODate('2025-06-19T12:40:57.000Z'), lastAppliedWallTime: ISODate('2025-06-19T12:40:57.228Z'), lastDurableWallTime: ISODate('2025-06-19T12:40:57.228Z'), lastHeartbeat: ISODate('2025-06-19T12:40:57.627Z'), lastHeartbeatRecv: ISODate('2025-06-19T12:40:57.927Z'), pingMs: Long('0'), lastHeartbeatMessage: '', syncSourceHost: 'mongodb-rxyipy-backup-mongodb-0.mongodb-rxyipy-backup-mongodb-headless.ns-azgwq.svc:27017', syncSourceId: 0, infoMessage: '', configVersion: 5, configTerm: 1 *** ], ok: 1, '$clusterTime': *** clusterTime: Timestamp(*** t: 1750336857, i: 1 ***), signature: *** hash: Binary.createFromBase64('Gceptg9xOM5DboSqABBn/fEeDLE=', 0), keyId: Long('7517639128301699075') *** ***, operationTime: Timestamp(*** t: 1750336857, i: 1 ***) *** mongodb-rxyipy-backup-mongodb [primary] admin> connect cluster Success delete cluster mongodb-rxyipy-backup `kbcli cluster delete mongodb-rxyipy-backup --auto-approve --namespace ns-azgwq ` Cluster mongodb-rxyipy-backup deleted pod_info:mongodb-rxyipy-backup-mongodb-0 3/3 Terminating 0 2m8s mongodb-rxyipy-backup-mongodb-1 3/3 Terminating 0 103s mongodb-rxyipy-backup-mongodb-2 3/3 Terminating 0 81s No resources found in ns-azgwq namespace. delete cluster pod done No resources found in ns-azgwq namespace. check cluster resource non-exist OK: pvc No resources found in ns-azgwq namespace. delete cluster done No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-azgwq-mongodb-rxyipy-20250619203834 --namespace ns-azgwq ` backup.dataprotection.kubeblocks.io/backup-ns-azgwq-mongodb-rxyipy-20250619203834 patched `kbcli cluster delete-backup mongodb-rxyipy --name backup-ns-azgwq-mongodb-rxyipy-20250619203834 --force --auto-approve --namespace ns-azgwq ` Backup backup-ns-azgwq-mongodb-rxyipy-20250619203834 deleted No opsrequests found in ns-azgwq namespace. cluster list-logs `kbcli cluster list-logs mongodb-rxyipy --namespace ns-azgwq ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update mongodb-rxyipy --enable-all-logs=true --namespace ns-azgwq Error from server (NotFound): pods "mongodb-rxyipy-mongodb-2" not found cluster logs `kbcli cluster logs mongodb-rxyipy --tail 30 --namespace ns-azgwq ` Defaulted container "mongodb" out of: mongodb, exporter, lorry, init-syncer (init) 2025-06-19T12:26:39Z INFO HA This member is Cluster's leader 2025-06-19T12:26:39Z DEBUG HA Refresh leader ttl 2025-06-19T12:27:39Z INFO HA This member is Cluster's leader 2025-06-19T12:27:39Z DEBUG HA Refresh leader ttl 2025-06-19T12:28:39Z INFO HA This member is Cluster's leader 2025-06-19T12:28:39Z DEBUG HA Refresh leader ttl 2025-06-19T12:29:39Z INFO HA This member is Cluster's leader 2025-06-19T12:29:39Z DEBUG HA Refresh leader ttl 2025-06-19T12:30:39Z INFO HA This member is Cluster's leader 2025-06-19T12:30:39Z DEBUG HA Refresh leader ttl 2025-06-19T12:31:39Z INFO HA This member is Cluster's leader 2025-06-19T12:31:39Z DEBUG HA Refresh leader ttl 2025-06-19T12:32:40Z INFO HA This member is Cluster's leader 2025-06-19T12:32:40Z DEBUG HA Refresh leader ttl 2025-06-19T12:33:41Z INFO HA This member is Cluster's leader 2025-06-19T12:33:42Z DEBUG HA Refresh leader ttl 2025-06-19T12:34:41Z INFO HA This member is Cluster's leader 2025-06-19T12:34:42Z DEBUG HA Refresh leader ttl 2025-06-19T12:35:42Z INFO HA This member is Cluster's leader 2025-06-19T12:35:42Z DEBUG HA Refresh leader ttl 2025-06-19T12:36:42Z INFO HA This member is Cluster's leader 2025-06-19T12:36:42Z DEBUG HA Refresh leader ttl 2025-06-19T12:37:42Z INFO HA This member is Cluster's leader 2025-06-19T12:37:42Z DEBUG HA Refresh leader ttl 2025-06-19T12:38:44Z INFO HA This member is Cluster's leader 2025-06-19T12:38:44Z DEBUG HA Refresh leader ttl 2025-06-19T12:39:44Z INFO HA This member is Cluster's leader 2025-06-19T12:39:44Z DEBUG HA Refresh leader ttl 2025-06-19T12:40:44Z INFO HA This member is Cluster's leader 2025-06-19T12:40:44Z DEBUG HA Refresh leader ttl cluster logs running `kbcli cluster logs mongodb-rxyipy --tail 30 --file-type=running --namespace ns-azgwq ` error: get pod component definition name in cluster.yaml fail delete cluster mongodb-rxyipy `kbcli cluster delete mongodb-rxyipy --auto-approve --namespace ns-azgwq ` Cluster mongodb-rxyipy deleted pod_info:mongodb-rxyipy-mongodb-0 3/3 Terminating 0 25m mongodb-rxyipy-mongodb-1 3/3 Terminating 0 25m mongodb-rxyipy-mongodb-2 3/3 Terminating 0 26m No resources found in ns-azgwq namespace. delete cluster pod done No resources found in ns-azgwq namespace. check cluster resource non-exist OK: pvc No resources found in ns-azgwq namespace. delete cluster done No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. No resources found in ns-azgwq namespace. Mongodb Test Suite All Done! --------------------------------------Mongodb (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentVersion=mongodb;ServiceVersion=6.0.16;]|[Description=Create a cluster and component version mongodb and service version 6.0.16] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[AddData]|[Values=egbjb]|[Description=Add data to the cluster] [PASSED]|[CheckAddDataReadonly]|[Values=egbjb;Role=Readonly]|[Description=Add data to the cluster readonly] [PASSED]|[Upgrade]|[ComponentName=mongodb;ComponentVersionFrom=6.0.16;ComponentVersionTo=6.0.20]|[Description=Upgrade the cluster specify component mongodb service version from 6.0.16 to 6.0.20] [PASSED]|[Upgrade]|[ComponentName=mongodb;ComponentVersionFrom=6.0.20;ComponentVersionTo=6.0.16]|[Description=Upgrade the cluster specify component mongodb service version from 6.0.20 to 6.0.16] [PASSED]|[HorizontalScaling Out]|[ComponentName=mongodb]|[Description=HorizontalScaling Out the cluster specify component mongodb] [PASSED]|[HorizontalScaling In]|[ComponentName=mongodb]|[Description=HorizontalScaling In the cluster specify component mongodb] [PASSED]|[Reconfiguring]|[ComponentName=mongodb;systemLog.quiet=true]|[Description=Reconfiguring the cluster specify component mongodb set systemLog.quiet=true] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=mongodb]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[SwitchOver]|[ComponentName=mongodb]|[Description=SwitchOver the cluster specify component mongodb] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[VerticalScaling]|[ComponentName=mongodb]|[Description=VerticalScaling the cluster specify component mongodb] [PASSED]|[VolumeExpansion]|[ComponentName=mongodb]|[Description=VolumeExpansion the cluster specify component mongodb] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=mongodb]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=mongodb]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=mongodb]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=mongodb]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=mongodb]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Check Monitor]|[Enable=true]|[Description=Check Monitor Enable] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=mongodb]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[RebuildInstance]|[ComponentName=mongodb]|[Description=Rebuild the cluster instance specify component mongodb] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=mongodb]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=mongodb]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=mongodb]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=mongodb]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[HscaleOfflineInstances]|[ComponentName=mongodb]|[Description=Hscale the cluster instances offline specify component mongodb] [PASSED]|[HscaleOnlineInstances]|[ComponentName=mongodb]|[Description=Hscale the cluster instances online specify component mongodb] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Reconfiguring]|[ComponentName=mongodb;systemLog.quiet=false,systemLog.verbosity=1]|[Description=Reconfiguring the cluster specify component mongodb set systemLog.quiet=false,systemLog.verbosity=1] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Backup] [PASSED]|[Restore]|[BackupMethod=volume-snapshot]|[Description=The cluster volume-snapshot Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=volume-snapshot]|[Description=Delete the volume-snapshot restore cluster] [PASSED]|[Backup]|[BackupMethod=datafile]|[Description=The cluster datafile Backup] [PASSED]|[Restore]|[BackupMethod=datafile]|[Description=The cluster datafile Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=datafile]|[Description=Delete the datafile restore cluster] [PASSED]|[RebuildInstance]|[ComponentName=mongodb]|[Description=Rebuild the cluster instance specify component mongodb] [PASSED]|[Backup]|[Schedule=true;BackupMethod=datafile]|[Description=The cluster Schedule datafile Backup] [PASSED]|[Restore]|[Schedule=true;BackupMethod=datafile]|[Description=The cluster Schedule datafile Restore] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[Schedule=true;BackupMethod=datafile]|[Description=Delete the Schedule datafile restore cluster] [PASSED]|[Backup]|[BackupMethod=dump]|[Description=The cluster dump Backup] [PASSED]|[Restore]|[BackupMethod=dump]|[Description=The cluster dump Restore] [PASSED]|[Check Data]|[BackupMethod=dump]|[Description=Check the cluster data restore via dump] [PASSED]|[Connect]|[ComponentName=mongodb]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=dump]|[Description=Delete the dump restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]