bash test/kbcli/test_kbcli_0.9.sh --type 32 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-grkwk ` `kubectl create namespace ns-grkwk` namespace/ns-grkwk created create namespace ns-grkwk done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 152M 0 --:--:-- --:--:-- --:--:-- 152M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed set component name:zookeeper set component version set component version:zookeeper set service versions:3.4.14,3.6.4,3.7.2,3.8.4,3.9.2 set service versions sorted:3.4.14,3.6.4,3.7.2,3.8.4,3.9.2 no cluster version found set zookeeper component definition set zookeeper component definition zookeeper-3 REPORT_COUNT 0:0 set replicas first:2,3.6.4|2,3.7.2|2,3.8.4|2,3.9.2 set replicas third:2,3.6.4 set replicas fourth:2,3.6.4 set minimum cmpv service version set minimum cmpv service version replicas:2,3.6.4 REPORT_COUNT:1 CLUSTER_TOPOLOGY: Error from server (NotFound): clusterdefinitions.apps.kubeblocks.io "zookeeper" not found Not found topology in cluster definition zookeeper LIMIT_CPU:0.5 LIMIT_MEMORY:1 storage size: 5 No resources found in ns-grkwk namespace. termination_policy:Halt create 2 replica Halt zookeeper cluster set component definition zookeeper-3, skip check component definition set component definition zookeeper-3, skip set component definition by component version apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: zookeep-qomiyr namespace: ns-grkwk spec: terminationPolicy: Halt componentSpecs: - name: zookeeper componentDef: zookeeper-3 replicas: 2 serviceVersion: 3.6.4 disableExporter: true resources: requests: cpu: 500m memory: 1Gi limits: cpu: 500m memory: 1Gi services: env: - name: ZOOKEEPER_IMAGE_VERSION value: "3.6.4" volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi - name: snapshot-log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi `kubectl apply -f test_create_zookeep-qomiyr.yaml` cluster.apps.kubeblocks.io/zookeep-qomiyr created apply test_create_zookeep-qomiyr.yaml Success `rm -rf test_create_zookeep-qomiyr.yaml` check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Sep 01,2025 11:19 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 snapshot-log:5Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=zookeep-qomiyr` No resources found in ns-grkwk namespace. Not found cluster secret DB_USERNAME:zookeeper;DB_PASSWORD:;DB_PORT:2181;DB_DATABASE: There is no password in Type: 32. describe cluster `kbcli cluster describe zookeep-qomiyr --namespace ns-grkwk ` Name: zookeep-qomiyr Created Time: Sep 01,2025 11:19 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-grkwk Running Halt Endpoints: COMPONENT MODE INTERNAL EXTERNAL zookeeper ReadWrite zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181 zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2888 zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:8080 zookeep-qomiyr-zookeeper-readable.ns-grkwk.svc.cluster.local:2181 zookeep-qomiyr-zookeeper-readable.ns-grkwk.svc.cluster.local:2888 zookeep-qomiyr-zookeeper-readable.ns-grkwk.svc.cluster.local:8080 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME zookeeper zookeep-qomiyr-zookeeper-0 follower Running 0 aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 zookeeper zookeep-qomiyr-zookeeper-1 leader Running 0 aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS zookeeper false 500m / 500m 1Gi / 1Gi data:5Gi default snapshot-log:5Gi default Images: COMPONENT TYPE IMAGE zookeeper docker.io/apecloud/zookeeper:3.6.4 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-grkwk zookeep-qomiyr `kbcli cluster label zookeep-qomiyr app.kubernetes.io/instance- --namespace ns-grkwk ` label "app.kubernetes.io/instance" not found. `kbcli cluster label zookeep-qomiyr app.kubernetes.io/instance=zookeep-qomiyr --namespace ns-grkwk ` `kbcli cluster label zookeep-qomiyr --list --namespace ns-grkwk ` NAME NAMESPACE LABELS zookeep-qomiyr ns-grkwk app.kubernetes.io/instance=zookeep-qomiyr label cluster app.kubernetes.io/instance=zookeep-qomiyr Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=zookeep-qomiyr --namespace ns-grkwk ` `kbcli cluster label zookeep-qomiyr --list --namespace ns-grkwk ` NAME NAMESPACE LABELS zookeep-qomiyr ns-grkwk app.kubernetes.io/instance=zookeep-qomiyr case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label zookeep-qomiyr case.name=kbcli.test2 --overwrite --namespace ns-grkwk ` `kbcli cluster label zookeep-qomiyr --list --namespace ns-grkwk ` NAME NAMESPACE LABELS zookeep-qomiyr ns-grkwk app.kubernetes.io/instance=zookeep-qomiyr case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label zookeep-qomiyr case.name- --namespace ns-grkwk ` `kbcli cluster label zookeep-qomiyr --list --namespace ns-grkwk ` NAME NAMESPACE LABELS zookeep-qomiyr ns-grkwk app.kubernetes.io/instance=zookeep-qomiyr delete cluster label case.name Success cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkServer.sh status' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` Defaulted container "zookeeper" out of: zookeeper, lorry, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file ZooKeeper JMX enabled by default Using config: /opt/bitnami/zookeeper/bin/../conf/zoo.cfg /opt/bitnami/java/bin/java Client port found: 2181. Client address: localhost. Client SSL: false. Mode: follower connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-zookeep-qomiyr" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-zookeep-qomiyr --namespace ns-grkwk ` Error from server (NotFound): pods "test-db-client-executionloop-zookeep-qomiyr" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-zookeep-qomiyr" not found `kubectl get secrets -l app.kubernetes.io/instance=zookeep-qomiyr` No resources found in ns-grkwk namespace. Not found cluster secret DB_USERNAME:zookeeper;DB_PASSWORD:;DB_PORT:2181;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-zookeep-qomiyr namespace: ns-grkwk spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local" - "--user" - "zookeeper" - "--password" - "" - "--port" - "2181" - "--dbtype" - "zookeeper" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-zookeep-qomiyr.yaml` pod/test-db-client-executionloop-zookeep-qomiyr created apply test-db-client-executionloop-zookeep-qomiyr.yaml Success `rm -rf test-db-client-executionloop-zookeep-qomiyr.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 0/1 ContainerCreating 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 0/1 ContainerCreating 0 9s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 0/1 ContainerCreating 0 14s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 0/1 ContainerCreating 0 19s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 24s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 50s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 55s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 60s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 65s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 71s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 76s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 81s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 86s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 91s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 96s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 101s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 106s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 1/1 Running 0 112s check pod test-db-client-executionloop-zookeep-qomiyr status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-zookeep-qomiyr 0/1 Completed 0 117s check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 snapshot-log:5Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.15.0-1092-azure 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.name=root 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.home=/root 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/app 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=915MB 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=15540MB 03:24:03.185 [main] INFO org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=972MB 03:24:03.187 [main] INFO org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181 sessionTimeout=5000 watcher=com.apecloud.dbtester.tester.ZookeeperTester$$Lambda$106/0x00000008400ea040@4f638935 03:24:03.190 [main] INFO o.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 03:24:03.194 [main] INFO o.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes 03:24:03.199 [main] INFO org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false 03:24:03.216 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:24:03.216 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:24:03.225 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.4:46628, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:24:03.236 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Session establishment complete on server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, session id = 0x100001334400000, negotiated timeout = 5000 Execution loop start: Path /executions_loop_path does not exist. Creating path... Path /executions_loop_path created successfully. Execution loop start: PUT /executions_loop_path "executions_loop_value_1" [ 1s ] executions total: 83 successful: 83 failed: 0 disconnect: 0 [ 2s ] executions total: 199 successful: 199 failed: 0 disconnect: 0 [ 3s ] executions total: 382 successful: 382 failed: 0 disconnect: 0 [ 4s ] executions total: 502 successful: 502 failed: 0 disconnect: 0 [ 5s ] executions total: 676 successful: 676 failed: 0 disconnect: 0 [ 6s ] executions total: 895 successful: 895 failed: 0 disconnect: 0 [ 7s ] executions total: 1107 successful: 1107 failed: 0 disconnect: 0 [ 8s ] executions total: 1395 successful: 1395 failed: 0 disconnect: 0 [ 9s ] executions total: 1552 successful: 1552 failed: 0 disconnect: 0 [ 10s ] executions total: 1690 successful: 1690 failed: 0 disconnect: 0 [ 11s ] executions total: 1951 successful: 1951 failed: 0 disconnect: 0 [ 12s ] executions total: 2296 successful: 2296 failed: 0 disconnect: 0 [ 13s ] executions total: 2488 successful: 2488 failed: 0 disconnect: 0 [ 14s ] executions total: 2749 successful: 2749 failed: 0 disconnect: 0 [ 15s ] executions total: 2836 successful: 2836 failed: 0 disconnect: 0 [ 16s ] executions total: 2968 successful: 2968 failed: 0 disconnect: 0 [ 17s ] executions total: 3078 successful: 3078 failed: 0 disconnect: 0 [ 18s ] executions total: 3273 successful: 3273 failed: 0 disconnect: 0 [ 19s ] executions total: 3535 successful: 3535 failed: 0 disconnect: 0 [ 20s ] executions total: 3886 successful: 3886 failed: 0 disconnect: 0 [ 21s ] executions total: 4228 successful: 4228 failed: 0 disconnect: 0 [ 22s ] executions total: 4554 successful: 4554 failed: 0 disconnect: 0 [ 23s ] executions total: 4867 successful: 4867 failed: 0 disconnect: 0 [ 24s ] executions total: 5200 successful: 5200 failed: 0 disconnect: 0 [ 25s ] executions total: 5488 successful: 5488 failed: 0 disconnect: 0 [ 26s ] executions total: 5682 successful: 5682 failed: 0 disconnect: 0 [ 27s ] executions total: 6018 successful: 6018 failed: 0 disconnect: 0 [ 28s ] executions total: 6369 successful: 6369 failed: 0 disconnect: 0 [ 29s ] executions total: 6443 successful: 6443 failed: 0 disconnect: 0 [ 30s ] executions total: 6764 successful: 6764 failed: 0 disconnect: 0 [ 31s ] executions total: 7137 successful: 7137 failed: 0 disconnect: 0 [ 32s ] executions total: 7278 successful: 7278 failed: 0 disconnect: 0 [ 33s ] executions total: 7506 successful: 7506 failed: 0 disconnect: 0 [ 34s ] executions total: 7804 successful: 7804 failed: 0 disconnect: 0 [ 35s ] executions total: 8200 successful: 8200 failed: 0 disconnect: 0 [ 36s ] executions total: 8593 successful: 8593 failed: 0 disconnect: 0 [ 37s ] executions total: 8767 successful: 8767 failed: 0 disconnect: 0 [ 38s ] executions total: 9170 successful: 9170 failed: 0 disconnect: 0 [ 39s ] executions total: 9571 successful: 9571 failed: 0 disconnect: 0 [ 40s ] executions total: 9957 successful: 9957 failed: 0 disconnect: 0 [ 41s ] executions total: 10235 successful: 10235 failed: 0 disconnect: 0 [ 42s ] executions total: 10422 successful: 10422 failed: 0 disconnect: 0 [ 43s ] executions total: 10681 successful: 10681 failed: 0 disconnect: 0 [ 44s ] executions total: 10925 successful: 10925 failed: 0 disconnect: 0 [ 45s ] executions total: 11124 successful: 11124 failed: 0 disconnect: 0 [ 46s ] executions total: 11485 successful: 11485 failed: 0 disconnect: 0 [ 47s ] executions total: 11853 successful: 11853 failed: 0 disconnect: 0 [ 48s ] executions total: 12192 successful: 12192 failed: 0 disconnect: 0 [ 49s ] executions total: 12572 successful: 12572 failed: 0 disconnect: 0 [ 50s ] executions total: 12855 successful: 12855 failed: 0 disconnect: 0 [ 51s ] executions total: 13057 successful: 13057 failed: 0 disconnect: 0 [ 52s ] executions total: 13313 successful: 13313 failed: 0 disconnect: 0 [ 53s ] executions total: 13592 successful: 13592 failed: 0 disconnect: 0 [ 54s ] executions total: 13990 successful: 13990 failed: 0 disconnect: 0 [ 55s ] executions total: 14385 successful: 14385 failed: 0 disconnect: 0 [ 56s ] executions total: 14700 successful: 14700 failed: 0 disconnect: 0 [ 57s ] executions total: 15089 successful: 15089 failed: 0 disconnect: 0 [ 58s ] executions total: 15485 successful: 15485 failed: 0 disconnect: 0 [ 60s ] executions total: 15674 successful: 15674 failed: 0 disconnect: 0 03:25:03.346 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100001334400000 closed Test Result: Total Executions: 15674 Successful Executions: 15674 Failed Executions: 0 Disconnection Counts: 0 Connection Information: 03:25:03.346 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100001334400000 Database Type: zookeeper Host: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local Port: 2181 Database: Table: User: zookeeper Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 15674 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-zookeep-qomiyr --namespace ns-grkwk ` pod/test-db-client-executionloop-zookeep-qomiyr patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-zookeep-qomiyr" force deleted test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-zookeep-qomiyr" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-zookeep-qomiyr --namespace ns-grkwk ` Error from server (NotFound): pods "test-db-client-connectionstress-zookeep-qomiyr" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-zookeep-qomiyr" not found `kubectl get secrets -l app.kubernetes.io/instance=zookeep-qomiyr` No resources found in ns-grkwk namespace. Not found cluster secret DB_USERNAME:zookeeper;DB_PASSWORD:;DB_PORT:2181;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-zookeep-qomiyr namespace: ns-grkwk spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local" - "--user" - "zookeeper" - "--password" - "" - "--port" - "2181" - "--database" - "" - "--dbtype" - "zookeeper" - "--test" - "connectionstress" - "--connections" - "500" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-zookeep-qomiyr.yaml` pod/test-db-client-connectionstress-zookeep-qomiyr created apply test-db-client-connectionstress-zookeep-qomiyr.yaml Success `rm -rf test-db-client-connectionstress-zookeep-qomiyr.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 9s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 50s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 61s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 66s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 71s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 76s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 81s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 86s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 91s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 97s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 102s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 1/1 Running 0 107s check pod test-db-client-connectionstress-zookeep-qomiyr status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-zookeep-qomiyr 0/1 Completed 0 112s check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 snapshot-log:5Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done 03:26:46.486 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:46.486 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:46.487 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:55948, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:46.488 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:48.090 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:48.090 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:48.091 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:55964, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:48.092 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:49.400 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:49.400 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:49.401 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:55972, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:49.402 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:51.105 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:51.106 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:51.109 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:55982, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:51.109 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:52.363 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:52.363 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:52.364 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:55988, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:52.364 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:53.467 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:53.467 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:53.468 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:55998, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:53.468 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:55.157 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:55.157 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:55.158 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:42798, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:55.158 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:56.692 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:56.692 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:56.694 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:42800, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:56.694 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:57.986 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:57.986 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:57.987 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:42816, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:57.987 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:26:59.557 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181. 03:26:59.557 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error) 03:26:59.558 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] INFO org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.244.1.1:42820, server: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181 03:26:59.558 [main-SendThread(zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local:2181)] WARN org.apache.zookeeper.ClientCnxn - Session 0x0 for sever zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local/10.0.198.163:2181, Closing socket connection. Attempting reconnect except it is a SessionExpiredException. org.apache.zookeeper.ClientCnxn$EndOfStreamException: Unable to read additional data from server sessionid 0x0, likely server has closed socket at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:77) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1290) 03:27:00.804 [main] INFO org.apache.zookeeper.ZooKeeper - Session: 0x100001334400001 closed Test Result: Failed to establish connection 499: Failed to execute command Connection stress test completed: Successful connections: 500 Duration: 70 seconds Connection Information: 03:27:00.804 [main-EventThread] INFO org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100001334400001 Database Type: zookeeper Host: zookeep-qomiyr-zookeeper.ns-grkwk.svc.cluster.local Port: 2181 Database: Table: User: zookeeper Org: Access Mode: mysql Test Type: connectionstress Connection Count: 500 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-zookeep-qomiyr --namespace ns-grkwk ` pod/test-db-client-connectionstress-zookeep-qomiyr patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-zookeep-qomiyr" force deleted check failover pod name failover pod name:zookeep-qomiyr-zookeeper-0 failover connectionstress Success endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=zookeep-qomiyr` No resources found in ns-grkwk namespace. Not found cluster secret DB_USERNAME:zookeeper;DB_PASSWORD:;DB_PORT:2181;DB_DATABASE: endpoints:localhost:2181 `echo "create /kbcli-test-data-qomiyr-lythm lythm" | bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181` Defaulted container "zookeeper" out of: zookeeper, lorry, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file Created /kbcli-test-data-qomiyr-lythm exec return message:/opt/bitnami/java/bin/java Connecting to localhost:2181 2025-09-01 03:28:47,962 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.6.4--d65253dcf68e9097c6e95a126463fd5fdeb4521c, built on 12/18/2022 18:10 GMT 2025-09-01 03:28:47,964 [myid:] - INFO [main:Environment@98] - Client environment:host.name=zookeep-qomiyr-zookeeper-0.zookeep-qomiyr-zookeeper-headless.ns-grkwk.svc.cluster.local 2025-09-01 03:28:47,964 [myid:] - INFO [main:Environment@98] - Client environment:java.version=11.0.18 2025-09-01 03:28:47,964 [myid:] - INFO [main:Environment@98] - Client environment:java.vendor=BellSoft 2025-09-01 03:28:47,965 [myid:] - INFO [main:Environment@98] - Client environment:java.home=/opt/bitnami/java 2025-09-01 03:28:47,965 [myid:] - INFO [main:Environment@98] - Client environment:java.class.path=/opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.8.4.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.36.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.36.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotspot-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.24.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-resolver-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-codec-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-3.2.5.jar:/opt/bitnami/zookeeper/bin/../lib/json-simple-1.1.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-ajax-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.13.4.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.4.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.4.jar:/opt/bitnami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.13.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/bitnami/zookeeper/bin/../conf: 2025-09-01 03:28:47,965 [myid:] - INFO [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2025-09-01 03:28:47,965 [myid:] - INFO [main:Environment@98] - Client environment:java.io.tmpdir=/tmp 2025-09-01 03:28:47,965 [myid:] - INFO [main:Environment@98] - Client environment:java.compiler= 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:os.name=Linux 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:os.arch=amd64 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:os.version=5.15.0-1092-azure 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:user.name=? 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:user.home=? 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:user.dir=/ 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.free=10MB 2025-09-01 03:28:47,966 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.max=247MB 2025-09-01 03:28:47,967 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.total=15MB 2025-09-01 03:28:47,970 [myid:] - INFO [main:ZooKeeper@1026] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@71248c21 2025-09-01 03:28:47,973 [myid:] - INFO [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2025-09-01 03:28:47,982 [myid:] - INFO [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes 2025-09-01 03:28:48,065 [myid:] - INFO [main:ClientCnxn@1741] - zookeeper.request.timeout value is 0. feature enabled=false Welcome to ZooKeeper! 2025-09-01 03:28:48,161 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1181] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. 2025-09-01 03:28:48,162 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1183] - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2025-09-01 03:28:48,170 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1013] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:43398, server: localhost/0:0:0:0:0:0:0:1:2181 JLine support is enabled 2025-09-01 03:28:48,262 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1452] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, session id = 0x13341c0004, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] create /kbcli-test-data-qomiyr-lythm lythm [zk: localhost:2181(CONNECTED) 1] 2025-09-01 03:28:48,575 [myid:] - INFO [main:ServiceUtils@45] - Exiting JVM with code 0 `echo "get /kbcli-test-data-qomiyr-lythm" | bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181` Defaulted container "zookeeper" out of: zookeeper, lorry, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg:/opt/bitnami/java/bin/java Connecting to localhost:2181 2025-09-01 03:28:50,077 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.6.4--d65253dcf68e9097c6e95a126463fd5fdeb4521c, built on 12/18/2022 18:10 GMT 2025-09-01 03:28:50,079 [myid:] - INFO [main:Environment@98] - Client environment:host.name=zookeep-qomiyr-zookeeper-0.zookeep-qomiyr-zookeeper-headless.ns-grkwk.svc.cluster.local 2025-09-01 03:28:50,080 [myid:] - INFO [main:Environment@98] - Client environment:java.version=11.0.18 2025-09-01 03:28:50,080 [myid:] - INFO [main:Environment@98] - Client environment:java.vendor=BellSoft 2025-09-01 03:28:50,080 [myid:] - INFO [main:Environment@98] - Client environment:java.home=/opt/bitnami/java 2025-09-01 03:28:50,160 [myid:] - INFO [main:Environment@98] - Client environment:java.class.path=/opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.8.4.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.36.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.36.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotspot-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.24.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-resolver-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-codec-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-3.2.5.jar:/opt/bitnami/zookeeper/bin/../lib/json-simple-1.1.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-ajax-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.13.4.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.4.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.4.jar:/opt/bitnami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.13.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/bitnami/zookeeper/bin/../conf: 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:java.io.tmpdir=/tmp 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:java.compiler= 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:os.name=Linux 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:os.arch=amd64 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:os.version=5.15.0-1092-azure 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:user.name=? 2025-09-01 03:28:50,161 [myid:] - INFO [main:Environment@98] - Client environment:user.home=? 2025-09-01 03:28:50,162 [myid:] - INFO [main:Environment@98] - Client environment:user.dir=/ 2025-09-01 03:28:50,162 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.free=10MB 2025-09-01 03:28:50,162 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.max=247MB 2025-09-01 03:28:50,162 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.total=15MB 2025-09-01 03:28:50,166 [myid:] - INFO [main:ZooKeeper@1026] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@71248c21 2025-09-01 03:28:50,170 [myid:] - INFO [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2025-09-01 03:28:50,181 [myid:] - INFO [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes 2025-09-01 03:28:50,265 [myid:] - INFO [main:ClientCnxn@1741] - zookeeper.request.timeout value is 0. feature enabled=false Welcome to ZooKeeper! 2025-09-01 03:28:50,278 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1181] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. 2025-09-01 03:28:50,278 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1183] - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2025-09-01 03:28:50,367 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1013] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:43412, server: localhost/0:0:0:0:0:0:0:1:2181 JLine support is enabled 2025-09-01 03:28:50,460 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1452] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, session id = 0x13341c0005, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] get /kbcli-test-data-qomiyr-lythm lythm [zk: localhost:2181(CONNECTED) 1] 2025-09-01 03:28:50,683 [myid:] - INFO [main:ServiceUtils@45] - Exiting JVM with code 0 create msg:[lythm] equal get msg:[lythm] `echo "delete /kbcli-test-data-qomiyr-lythm" | bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181` Defaulted container "zookeeper" out of: zookeeper, lorry, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message:/opt/bitnami/java/bin/java Connecting to localhost:2181 2025-09-01 03:28:52,263 [myid:] - INFO [main:Environment@98] - Client environment:zookeeper.version=3.6.4--d65253dcf68e9097c6e95a126463fd5fdeb4521c, built on 12/18/2022 18:10 GMT 2025-09-01 03:28:52,266 [myid:] - INFO [main:Environment@98] - Client environment:host.name=zookeep-qomiyr-zookeeper-0.zookeep-qomiyr-zookeeper-headless.ns-grkwk.svc.cluster.local 2025-09-01 03:28:52,266 [myid:] - INFO [main:Environment@98] - Client environment:java.version=11.0.18 2025-09-01 03:28:52,266 [myid:] - INFO [main:Environment@98] - Client environment:java.vendor=BellSoft 2025-09-01 03:28:52,266 [myid:] - INFO [main:Environment@98] - Client environment:java.home=/opt/bitnami/java 2025-09-01 03:28:52,266 [myid:] - INFO [main:Environment@98] - Client environment:java.class.path=/opt/bitnami/zookeeper/bin/../zookeeper-server/target/classes:/opt/bitnami/zookeeper/bin/../build/classes:/opt/bitnami/zookeeper/bin/../zookeeper-server/target/lib/*.jar:/opt/bitnami/zookeeper/bin/../build/lib/*.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-prometheus-metrics-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-jute-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/zookeeper-3.6.4.jar:/opt/bitnami/zookeeper/bin/../lib/snappy-java-1.1.8.4.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-reload4j-1.7.36.jar:/opt/bitnami/zookeeper/bin/../lib/slf4j-api-1.7.36.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_servlet-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_hotspot-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient_common-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/simpleclient-0.6.0.jar:/opt/bitnami/zookeeper/bin/../lib/reload4j-1.2.24.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-unix-common-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-native-epoll-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-classes-epoll-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-transport-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-resolver-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-handler-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-common-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-codec-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/netty-buffer-4.1.86.Final.jar:/opt/bitnami/zookeeper/bin/../lib/metrics-core-3.2.5.jar:/opt/bitnami/zookeeper/bin/../lib/json-simple-1.1.1.jar:/opt/bitnami/zookeeper/bin/../lib/jline-2.14.6.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-ajax-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-util-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-servlet-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-server-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-security-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-io-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/jetty-http-9.4.49.v20220914.jar:/opt/bitnami/zookeeper/bin/../lib/javax.servlet-api-3.1.0.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-databind-2.13.4.2.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-core-2.13.4.jar:/opt/bitnami/zookeeper/bin/../lib/jackson-annotations-2.13.4.jar:/opt/bitnami/zookeeper/bin/../lib/commons-cli-1.4.jar:/opt/bitnami/zookeeper/bin/../lib/audience-annotations-0.13.0.jar:/opt/bitnami/zookeeper/bin/../zookeeper-*.jar:/opt/bitnami/zookeeper/bin/../zookeeper-server/src/main/resources/lib/*.jar:/opt/bitnami/zookeeper/bin/../conf: 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:java.io.tmpdir=/tmp 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:java.compiler= 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:os.name=Linux 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:os.arch=amd64 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:os.version=5.15.0-1092-azure 2025-09-01 03:28:52,267 [myid:] - INFO [main:Environment@98] - Client environment:user.name=? 2025-09-01 03:28:52,268 [myid:] - INFO [main:Environment@98] - Client environment:user.home=? 2025-09-01 03:28:52,268 [myid:] - INFO [main:Environment@98] - Client environment:user.dir=/ 2025-09-01 03:28:52,268 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.free=10MB 2025-09-01 03:28:52,268 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.max=247MB 2025-09-01 03:28:52,268 [myid:] - INFO [main:Environment@98] - Client environment:os.memory.total=15MB 2025-09-01 03:28:52,271 [myid:] - INFO [main:ZooKeeper@1026] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@71248c21 2025-09-01 03:28:52,275 [myid:] - INFO [main:X509Util@77] - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation 2025-09-01 03:28:52,369 [myid:] - INFO [main:ClientCnxnSocket@239] - jute.maxbuffer value is 1048575 Bytes 2025-09-01 03:28:52,375 [myid:] - INFO [main:ClientCnxn@1741] - zookeeper.request.timeout value is 0. feature enabled=false Welcome to ZooKeeper! 2025-09-01 03:28:52,460 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1181] - Opening socket connection to server localhost/127.0.0.1:2181. 2025-09-01 03:28:52,461 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1183] - SASL config status: Will not attempt to authenticate using SASL (unknown error) 2025-09-01 03:28:52,469 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1013] - Socket connection established, initiating session, client: /127.0.0.1:36668, server: localhost/127.0.0.1:2181 JLine support is enabled 2025-09-01 03:28:52,562 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1452] - Session establishment complete on server localhost/127.0.0.1:2181, session id = 0x13341c0006, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null [zk: localhost:2181(CONNECTED) 0] delete /kbcli-test-data-qomiyr-lythm [zk: localhost:2181(CONNECTED) 1] 2025-09-01 03:28:52,866 [myid:] - INFO [main:ServiceUtils@45] - Exiting JVM with code 0 cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart zookeep-qomiyr --auto-approve --force=true --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-restart-4sbvx created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-restart-4sbvx -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-restart-4sbvx ns-grkwk Restart zookeep-qomiyr zookeeper Pending -/- Sep 01,2025 11:28 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster zookeep-qomiyr yaml-------------------------------------- `kubectl get cluster zookeep-qomiyr -o yaml --namespace ns-grkwk ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"zookeep-qomiyr-restart-4sbvx","type":"Restart"***]' kubeblocks.io/reconcile: "2025-09-01T03:35:02.712732694Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"zookeep-qomiyr","namespace":"ns-grkwk"***,"spec":***"componentSpecs":[***"componentDef":"zookeeper-3","disableExporter":true,"env":[***"name":"ZOOKEEPER_IMAGE_VERSION","value":"3.6.4"***],"name":"zookeeper","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"1Gi"***,"requests":***"cpu":"500m","memory":"1Gi"***,"serviceVersion":"3.6.4","services":null,"volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"5Gi"***,"storageClassName":null***,***"name":"snapshot-log","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"5Gi"***,"storageClassName":null***]***],"terminationPolicy":"Halt"*** creationTimestamp: "2025-09-01T03:19:28Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 2 labels: app.kubernetes.io/instance: zookeep-qomiyr name: zookeep-qomiyr namespace: ns-grkwk resourceVersion: "25566" uid: fc634d34-0969-4aa4-ad46-30a0c1ce24c9 spec: componentSpecs: - componentDef: zookeeper-3 disableExporter: true env: - name: ZOOKEEPER_IMAGE_VERSION value: 3.6.4 name: zookeeper replicas: 2 resources: limits: cpu: 500m memory: 1Gi requests: cpu: 500m memory: 1Gi serviceVersion: 3.6.4 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi - name: snapshot-log spec: accessModes: - ReadWriteOnce resources: requests: storage: 5Gi resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Halt status: components: zookeeper: phase: Updating podsReady: false podsReadyTime: "2025-09-01T03:23:14Z" conditions: - lastTransitionTime: "2025-09-01T03:19:28Z" message: 'The operator has started the provisioning of Cluster: zookeep-qomiyr' observedGeneration: 2 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:19:28Z" message: Successfully applied for resources observedGeneration: 2 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T03:28:54Z" message: 'pods are not ready in Components: [zookeeper], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T03:28:54Z" message: 'pods are unavailable in Components: [zookeeper], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 2 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster zookeep-qomiyr-------------------------------------- `kubectl describe cluster zookeep-qomiyr --namespace ns-grkwk ` Name: zookeep-qomiyr Namespace: ns-grkwk Labels: app.kubernetes.io/instance=zookeep-qomiyr Annotations: kubeblocks.io/ops-request: [***"name":"zookeep-qomiyr-restart-4sbvx","type":"Restart"***] kubeblocks.io/reconcile: 2025-09-01T03:35:02.712732694Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:19:28Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 2 Resource Version: 25566 UID: fc634d34-0969-4aa4-ad46-30a0c1ce24c9 Spec: Component Specs: Component Def: zookeeper-3 Disable Exporter: true Env: Name: ZOOKEEPER_IMAGE_VERSION Value: 3.6.4 Name: zookeeper Replicas: 2 Resources: Limits: Cpu: 500m Memory: 1Gi Requests: Cpu: 500m Memory: 1Gi Service Version: 3.6.4 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 5Gi Name: snapshot-log Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 5Gi Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Halt Status: Components: Zookeeper: Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T03:23:14Z Conditions: Last Transition Time: 2025-09-01T03:19:28Z Message: The operator has started the provisioning of Cluster: zookeep-qomiyr Observed Generation: 2 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:19:28Z Message: Successfully applied for resources Observed Generation: 2 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T03:28:54Z Message: pods are not ready in Components: [zookeeper], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T03:28:54Z Message: pods are unavailable in Components: [zookeeper], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 2 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 16m cluster-controller component is Creating Normal ComponentPhaseTransition 12m cluster-controller component is Running Normal AllReplicasReady 12m cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 12m cluster-controller Cluster: zookeep-qomiyr is ready, current phase is Running Normal Running 12m cluster-controller Cluster: zookeep-qomiyr is ready, current phase is Running Normal PreCheckSucceed 7m20s (x2 over 16m) cluster-controller The operator has started the provisioning of Cluster: zookeep-qomiyr Normal ApplyResourcesSucceed 7m20s (x2 over 16m) cluster-controller Successfully applied for resources Normal ComponentPhaseTransition 7m19s cluster-controller component is Updating Warning ReplicasNotReady 7m19s cluster-controller pods are not ready in Components: [zookeeper], refer to related component message in Cluster.status.components Warning ComponentsNotReady 7m19s cluster-controller pods are unavailable in Components: [zookeeper], refer to related component message in Cluster.status.components Warning FailedAttachVolume 71s (x3 over 2m48s) event-controller Pod zookeep-qomiyr-zookeeper-1: AttachVolume.Attach failed for volume "pvc-1f91e0f3-c74e-4416-b430-97e2ad3d0dd5" : volume attachment is being deleted Warning FailedAttachVolume 71s (x3 over 2m48s) event-controller Pod zookeep-qomiyr-zookeeper-1: AttachVolume.Attach failed for volume "pvc-995590a8-cbf5-4024-8d9e-fc398b95ff8e" : volume attachment is being deleted ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:28 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Init:0/1 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:30 UTC+0800 snapshot-log:5Gi pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 pod_status:Init:0/1 check pod status done check cluster status again check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-restart-4sbvx ns-grkwk Restart zookeep-qomiyr zookeeper Running 0/2 Sep 01,2025 11:28 UTC+0800 ops_status:zookeep-qomiyr-restart-4sbvx ns-grkwk Restart zookeep-qomiyr zookeeper Running 0/2 Sep 01,2025 11:28 UTC+0800 ops_status:zookeep-qomiyr-restart-4sbvx ns-grkwk Restart zookeep-qomiyr zookeeper Running 1/2 Sep 01,2025 11:28 UTC+0800 check ops status done ops_status:zookeep-qomiyr-restart-4sbvx ns-grkwk Restart zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-restart-4sbvx --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-restart-4sbvx patched `kbcli cluster delete-ops --name zookeep-qomiyr-restart-4sbvx --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-restart-4sbvx deleted endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success cluster update enable-all-logs `kbcli cluster update zookeep-qomiyr --enable-all-logs=true --namespace ns-grkwk ` error: name is required check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:28 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 500m / 500m 1Gi / 1Gi data:5Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:30 UTC+0800 snapshot-log:5Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale zookeep-qomiyr --auto-approve --force=true --components zookeeper --cpu 600m --memory 1.1Gi --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-verticalscaling-8bwmx created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-verticalscaling-8bwmx -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-verticalscaling-8bwmx ns-grkwk VerticalScaling zookeep-qomiyr zookeeper Running -/- Sep 01,2025 11:41 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 600m / 600m 1181116006400m / 1181116006400m data:5Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:41 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 600m / 600m 1181116006400m / 1181116006400m data:5Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:42 UTC+0800 snapshot-log:5Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-verticalscaling-8bwmx ns-grkwk VerticalScaling zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:41 UTC+0800 check ops status done ops_status:zookeep-qomiyr-verticalscaling-8bwmx ns-grkwk VerticalScaling zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-verticalscaling-8bwmx --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-verticalscaling-8bwmx patched `kbcli cluster delete-ops --name zookeep-qomiyr-verticalscaling-8bwmx --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-verticalscaling-8bwmx deleted endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success skip cluster HorizontalScaling `kubectl get pvc -l app.kubernetes.io/instance=zookeep-qomiyr,apps.kubeblocks.io/component-name=zookeeper,apps.kubeblocks.io/vct-name=data --namespace ns-grkwk ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in zookeep-qomiyr namespace. `kbcli cluster volume-expand zookeep-qomiyr --auto-approve --force=true --components zookeeper --volume-claim-templates data --storage 10Gi --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-volumeexpansion-bbdx8 created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-volumeexpansion-bbdx8 -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-volumeexpansion-bbdx8 ns-grkwk VolumeExpansion zookeep-qomiyr zookeeper Running -/- Sep 01,2025 11:43 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:41 UTC+0800 snapshot-log:5Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:42 UTC+0800 snapshot-log:5Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done No resources found in zookeep-qomiyr namespace. check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-volumeexpansion-bbdx8 ns-grkwk VolumeExpansion zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:43 UTC+0800 check ops status done ops_status:zookeep-qomiyr-volumeexpansion-bbdx8 ns-grkwk VolumeExpansion zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-volumeexpansion-bbdx8 --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-volumeexpansion-bbdx8 patched `kbcli cluster delete-ops --name zookeep-qomiyr-volumeexpansion-bbdx8 --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-volumeexpansion-bbdx8 deleted endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=zookeep-qomiyr,apps.kubeblocks.io/component-name=zookeeper,apps.kubeblocks.io/vct-name=snapshot-log --namespace ns-grkwk ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in zookeep-qomiyr namespace. `kbcli cluster volume-expand zookeep-qomiyr --auto-approve --force=true --components zookeeper --volume-claim-templates snapshot-log --storage 7Gi --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-volumeexpansion-sfzzm created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-volumeexpansion-sfzzm -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-volumeexpansion-sfzzm ns-grkwk VolumeExpansion zookeep-qomiyr zookeeper Pending -/- Sep 01,2025 11:47 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:41 UTC+0800 snapshot-log:7Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:42 UTC+0800 snapshot-log:7Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done No resources found in zookeep-qomiyr namespace. check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-volumeexpansion-sfzzm ns-grkwk VolumeExpansion zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:47 UTC+0800 check ops status done ops_status:zookeep-qomiyr-volumeexpansion-sfzzm ns-grkwk VolumeExpansion zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-volumeexpansion-sfzzm --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-volumeexpansion-sfzzm patched `kbcli cluster delete-ops --name zookeep-qomiyr-volumeexpansion-sfzzm --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-volumeexpansion-sfzzm deleted endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success cluster configure component_tmp: zookeeper check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure zookeep-qomiyr --auto-approve --force=true --set syncLimit=40 --components zookeeper --config-spec zookeeper-config --config-file zoo.cfg --namespace ns-grkwk ` Will updated configure file meta: ConfigSpec: zookeeper-config ConfigFile: zoo.cfg ComponentName: zookeeper ClusterName: zookeep-qomiyr OpsRequest zookeep-qomiyr-reconfiguring-ngmxq created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-reconfiguring-ngmxq -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-reconfiguring-ngmxq ns-grkwk Reconfiguring zookeep-qomiyr zookeeper Pending -/- Sep 01,2025 11:52 UTC+0800 ops_status:zookeep-qomiyr-reconfiguring-ngmxq ns-grkwk Reconfiguring zookeep-qomiyr zookeeper Creating -/- Sep 01,2025 11:52 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:52 UTC+0800 snapshot-log:7Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:57 UTC+0800 snapshot-log:7Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-reconfiguring-ngmxq ns-grkwk Reconfiguring zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:52 UTC+0800 check ops status done ops_status:zookeep-qomiyr-reconfiguring-ngmxq ns-grkwk Reconfiguring zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-reconfiguring-ngmxq --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-reconfiguring-ngmxq patched `kbcli cluster delete-ops --name zookeep-qomiyr-reconfiguring-ngmxq --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-reconfiguring-ngmxq deleted check config variables Defaulted container "zookeeper" out of: zookeeper, lorry, init-lorry (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 40 configure:[syncLimit] result actual:[40] equal expected:[40] endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop zookeep-qomiyr --auto-approve --force=true --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-stop-ckwwk created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-stop-ckwwk -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-stop-ckwwk ns-grkwk Stop zookeep-qomiyr Sep 01,2025 11:58 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Stopped Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-stop-ckwwk ns-grkwk Stop zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:58 UTC+0800 check ops status done ops_status:zookeep-qomiyr-stop-ckwwk ns-grkwk Stop zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:58 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-stop-ckwwk --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-stop-ckwwk patched `kbcli cluster delete-ops --name zookeep-qomiyr-stop-ckwwk --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-stop-ckwwk deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start zookeep-qomiyr --force=true --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-start-zbtck created successfully, you can view the progress: kbcli cluster describe-ops zookeep-qomiyr-start-zbtck -n ns-grkwk check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-start-zbtck ns-grkwk Start zookeep-qomiyr Creating -/- Sep 01,2025 11:58 UTC+0800 check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk Halt Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:58 UTC+0800 snapshot-log:7Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:58 UTC+0800 snapshot-log:7Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done check ops status `kbcli cluster list-ops zookeep-qomiyr --status all --namespace ns-grkwk ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME zookeep-qomiyr-start-zbtck ns-grkwk Start zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:58 UTC+0800 check ops status done ops_status:zookeep-qomiyr-start-zbtck ns-grkwk Start zookeep-qomiyr zookeeper Succeed 2/2 Sep 01,2025 11:58 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests zookeep-qomiyr-start-zbtck --namespace ns-grkwk ` opsrequest.apps.kubeblocks.io/zookeep-qomiyr-start-zbtck patched `kbcli cluster delete-ops --name zookeep-qomiyr-start-zbtck --force --auto-approve --namespace ns-grkwk ` OpsRequest zookeep-qomiyr-start-zbtck deleted endpoints:localhost:2181 check db_client batch data count `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181 get /executions_loop_path' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- sh ` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update zookeep-qomiyr --termination-policy=WipeOut --namespace ns-grkwk ` cluster.apps.kubeblocks.io/zookeep-qomiyr updated check cluster status `kbcli cluster list zookeep-qomiyr --show-labels --namespace ns-grkwk ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS zookeep-qomiyr ns-grkwk WipeOut Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=zookeep-qomiyr check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances zookeep-qomiyr --namespace ns-grkwk ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME zookeep-qomiyr-zookeeper-0 ns-grkwk zookeep-qomiyr zookeeper Running follower 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:58 UTC+0800 snapshot-log:7Gi zookeep-qomiyr-zookeeper-1 ns-grkwk zookeep-qomiyr zookeeper Running leader 0 600m / 600m 1181116006400m / 1181116006400m data:10Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 11:58 UTC+0800 snapshot-log:7Gi check pod status done endpoints:localhost:2181 check cluster connect `echo 'bash /opt/bitnami/zookeeper/bin/zkCli.sh -server localhost:2181' | kubectl exec -it zookeep-qomiyr-zookeeper-0 --namespace ns-grkwk -- bash` check cluster connect done cluster list-logs `kbcli cluster list-logs zookeep-qomiyr --namespace ns-grkwk ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update zookeep-qomiyr --enable-all-logs=true --namespace ns-grkwk Error from server (NotFound): pods "zookeep-qomiyr-zookeeper-0" not found cluster logs `kbcli cluster logs zookeep-qomiyr --tail 30 --namespace ns-grkwk ` Defaulted container "zookeeper" out of: zookeeper, lorry, init-lorry (init) 2025-09-01 03:59:02,046 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Leader@581] - LEADING - LEADER ELECTION TOOK - 205 MS 2025-09-01 03:59:02,046 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@864] - Peer state changed: leading - discovery 2025-09-01 03:59:02,046 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):FileTxnSnapLog@479] - Snapshotting: 0x700000002 to /bitnami/zookeeper/data/version-2/snapshot.700000002 2025-09-01 03:59:02,047 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ZooKeeperServer@540] - Snapshot taken in 1 ms 2025-09-01 03:59:02,053 [myid:1] - INFO [LearnerHandler-/10.244.1.211:33112:LearnerHandler@504] - Follower sid: 0 : info : zookeep-qomiyr-zookeeper-0.zookeep-qomiyr-zookeeper-headless.ns-grkwk.svc.cluster.local:2888:3888:participant 2025-09-01 03:59:02,062 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@1820] - Dynamic reconfig is disabled, we don't store the last seen config. 2025-09-01 03:59:02,069 [myid:1] - INFO [LearnerHandler-/10.244.1.211:33112:ZKDatabase@345] - On disk txn sync enabled with snapshotSizeFactor 0.33 2025-09-01 03:59:02,070 [myid:1] - INFO [LearnerHandler-/10.244.1.211:33112:LearnerHandler@807] - Synchronizing with Learner sid: 0 maxCommittedLog=0x700000002 minCommittedLog=0x700000001 lastProcessedZxid=0x700000002 peerLastZxid=0x700000002 2025-09-01 03:59:02,070 [myid:1] - INFO [LearnerHandler-/10.244.1.211:33112:LearnerHandler@852] - Sending DIFF zxid=0x700000002 for peer sid: 0 2025-09-01 03:59:02,076 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@864] - Peer state changed: leading - synchronization 2025-09-01 03:59:02,098 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Leader@1504] - Have quorum of supporters, sids: [[0, 1]]; starting up and setting last processed zxid: 0x800000000 2025-09-01 03:59:02,098 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):Leader@1526] - Dynamic reconfig feature is disabled, skip designatedLeader calculation and reconfig processing. 2025-09-01 03:59:02,104 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@475] - Configuring CommitProcessor with readBatchSize -1 commitBatchSize 1 2025-09-01 03:59:02,104 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):CommitProcessor@437] - Configuring CommitProcessor with 1 worker threads. 2025-09-01 03:59:02,109 [myid:1] - INFO [ProcessThread(sid:1 cport:-1)::PrepRequestProcessor@136] - PrepRequestProcessor (sid:1) started, reconfigEnabled=false 2025-09-01 03:59:02,110 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):ContainerManager@83] - Using checkIntervalMs=60000 maxPerMinute=10000 maxNeverUsedIntervalMs=0 2025-09-01 03:59:02,111 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):RequestThrottler@74] - zookeeper.request_throttler.shutdownTimeout = 10000 2025-09-01 03:59:02,122 [myid:1] - INFO [QuorumPeer[myid=1](plain=[0:0:0:0:0:0:0:0]:2181)(secure=disabled):QuorumPeer@864] - Peer state changed: leading - broadcast 2025-09-01 03:59:07,537 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@515] - Processing ruok command from /0:0:0:0:0:0:0:1:41180 2025-09-01 03:59:07,652 [myid:1] - INFO [NIOWorkerThread-2:NIOServerCnxn@515] - Processing srvr command from /0:0:0:0:0:0:0:1:41190 2025-09-01 03:59:17,314 [myid:1] - INFO [SyncThread:1:FileTxnLog@284] - Creating new log file: log.800000001 2025-09-01 03:59:17,325 [myid:1] - INFO [CommitProcessor:1:LeaderSessionTracker@104] - Committing global session 0x238b800000 2025-09-01 03:59:17,538 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@515] - Processing ruok command from /0:0:0:0:0:0:0:1:34160 2025-09-01 03:59:17,538 [myid:1] - INFO [NIOWorkerThread-2:NIOServerCnxn@515] - Processing ruok command from /0:0:0:0:0:0:0:1:34172 2025-09-01 03:59:17,652 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@515] - Processing srvr command from /0:0:0:0:0:0:0:1:34186 2025-09-01 03:59:19,921 [myid:1] - INFO [CommitProcessor:1:LeaderSessionTracker@104] - Committing global session 0x238b800001 2025-09-01 03:59:27,403 [myid:1] - INFO [CommitProcessor:1:LeaderSessionTracker@104] - Committing global session 0x238b800002 2025-09-01 03:59:27,536 [myid:1] - INFO [NIOWorkerThread-2:NIOServerCnxn@515] - Processing ruok command from /0:0:0:0:0:0:0:1:47688 2025-09-01 03:59:27,537 [myid:1] - INFO [NIOWorkerThread-2:NIOServerCnxn@515] - Processing ruok command from /0:0:0:0:0:0:0:1:47690 2025-09-01 03:59:27,652 [myid:1] - INFO [NIOWorkerThread-1:NIOServerCnxn@515] - Processing srvr command from /0:0:0:0:0:0:0:1:47698 delete cluster zookeep-qomiyr `kbcli cluster delete zookeep-qomiyr --auto-approve --namespace ns-grkwk ` Cluster zookeep-qomiyr deleted pod_info:zookeep-qomiyr-zookeeper-0 2/2 Running 0 68s zookeep-qomiyr-zookeeper-1 2/2 Running 0 68s No resources found in ns-grkwk namespace. delete cluster pod done No resources found in ns-grkwk namespace. check cluster resource non-exist OK: pvc No resources found in ns-grkwk namespace. delete cluster done No resources found in ns-grkwk namespace. No resources found in ns-grkwk namespace. No resources found in ns-grkwk namespace. Zookeeper Test Suite All Done! --------------------------------------Zookeeper (Topology = Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentVersion=zookeeper;ServiceVersion=3.6.4;]|[Description=Create a cluster and component version zookeeper and service version 3.6.4] [PASSED]|[Connect]|[ComponentName=zookeeper]|[Description=Connect to the cluster] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=zookeeper]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Update]|[Logs=Enable]|[Description=Update the cluster Logs enable] [PASSED]|[VerticalScaling]|[ComponentName=zookeeper]|[Description=VerticalScaling the cluster specify component zookeeper] [PASSED]|[VolumeExpansion]|[ComponentName=zookeeper;ComponentVolume=data]|[Description=VolumeExpansion the cluster specify component zookeeper and volume data] [PASSED]|[VolumeExpansion]|[ComponentName=zookeeper;ComponentVolume=snapshot-log]|[Description=VolumeExpansion the cluster specify component zookeeper and volume snapshot-log] [PASSED]|[Reconfiguring]|[ComponentName=zookeeper;syncLimit=40]|[Description=Reconfiguring the cluster specify component zookeeper set syncLimit=40] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]