source commons files source engines files source kubeblocks files source kubedb files CLUSTER_NAME: `kubectl get namespace | grep ns-keddj ` `kubectl create namespace ns-keddj` namespace/ns-keddj created create namespace ns-keddj done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 56.8M 0 --:--:-- --:--:-- --:--:-- 56.8M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.1 done Kubernetes: v1.32.6 KubeBlocks: 1.0.1 kbcli: 1.0.1 Kubernetes Env: v1.32.6 check snapshot controller check snapshot controller done POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default KubeBlocks version is:1.0.1 skip upgrade KubeBlocks current KubeBlocks version: 1.0.1 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:clickhouse set component version set component version:clickhouse set service versions:25.4.4,24.8.3,22.8.21,22.3.20,22.3.18 set service versions sorted:22.3.18,22.3.20,22.8.21,24.8.3,25.4.4 set clickhouse component definition set clickhouse component definition clickhouse-keeper-1.0.1 REPORT_COUNT 0:0 set replicas first:2,22.3.18|2,22.3.20|2,22.8.21|2,24.8.3|2,25.4.4 set replicas third:2,22.8.21 set replicas fourth:2,22.8.21 set minimum cmpv service version set minimum cmpv service version replicas:2,22.8.21 REPORT_COUNT:1 CLUSTER_TOPOLOGY:cluster topology cluster found in cluster definition clickhouse set clickhouse component definition set clickhouse component definition clickhouse-keeper-1.0.1 LIMIT_CPU:0.2 LIMIT_MEMORY:2 storage size: 20 CLUSTER_NAME:clkhouse-ccrhgq No resources found in ns-keddj namespace. pod_info: termination_policy:DoNotTerminate create 2 replica DoNotTerminate clickhouse cluster check component definition set component definition by component version check cmpd by labels check cmpd by compDefs set component definition: clickhouse-1.0.1 by component version:clickhouse apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: clkhouse-ccrhgq namespace: ns-keddj spec: clusterDef: clickhouse topology: cluster terminationPolicy: DoNotTerminate componentSpecs: - name: ch-keeper serviceVersion: 22.8.21 replicas: 1 disableExporter: false services: - name: default serviceType: ClusterIP systemAccounts: - name: admin passwordConfig: length: 10 numDigits: 5 numSymbols: 0 letterCase: MixedCases seed: clkhouse-ccrhgq resources: requests: cpu: 200m memory: 2Gi limits: cpu: 200m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi shardings: - name: clickhouse shards: 2 template: name: clickhouse serviceVersion: 22.8.21 env: - name: "INIT_CLUSTER_NAME" value: "default" replicas: 2 disableExporter: false services: - name: default serviceType: ClusterIP systemAccounts: - name: admin passwordConfig: length: 10 numDigits: 5 numSymbols: 0 letterCase: MixedCases seed: clkhouse-ccrhgq resources: requests: cpu: 200m memory: 2Gi limits: cpu: 200m memory: 2Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi `kubectl apply -f test_create_clkhouse-ccrhgq.yaml` cluster.apps.kubeblocks.io/clkhouse-ccrhgq created apply test_create_clkhouse-ccrhgq.yaml Success `rm -rf test_create_clkhouse-ccrhgq.yaml` check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Creating Sep 11,2025 20:24 UTC+0800 clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:26 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:26 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:26 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:26 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default check pod clkhouse-ccrhgq-clickhouse-4wc-0 container_name clickhouse exist password 7M6Rq5c2x0 check pod clkhouse-ccrhgq-clickhouse-4wc-0 container_name kbagent exist password 7M6Rq5c2x0 No container logs contain secret password. describe cluster `kbcli cluster describe clkhouse-ccrhgq --namespace ns-keddj ` Name: clkhouse-ccrhgq Created Time: Sep 11,2025 20:24 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-keddj clickhouse cluster Running DoNotTerminate Endpoints: COMPONENT INTERNAL EXTERNAL ch-keeper clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:8123 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:8443 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9000 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9009 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9010 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:8001 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9181 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9234 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9281 clkhouse-ccrhgq-ch-keeper.ns-keddj.svc.cluster.local:9440 clickhouse(clickhouse-4wc) clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:8001 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:8123 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:8443 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9000 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9004 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9005 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9009 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9010 clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9440 clickhouse(clickhouse-c94) clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:8001 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:8123 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:8443 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:9000 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:9004 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:9005 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:9009 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:9010 clkhouse-ccrhgq-clickhouse-c94.ns-keddj.svc.cluster.local:9440 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME ch-keeper 22.8.21 clkhouse-ccrhgq-ch-keeper-0 leader Running 0 aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clickhouse(clickhouse-4wc) 22.8.21 clkhouse-ccrhgq-clickhouse-4wc-0 Running 0 aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:26 UTC+0800 clickhouse(clickhouse-4wc) 22.8.21 clkhouse-ccrhgq-clickhouse-4wc-1 Running 0 aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:26 UTC+0800 clickhouse(clickhouse-c94) 22.8.21 clkhouse-ccrhgq-clickhouse-c94-0 Running 0 aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:26 UTC+0800 clickhouse(clickhouse-c94) 22.8.21 clkhouse-ccrhgq-clickhouse-c94-1 Running 0 aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:26 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS ch-keeper 200m / 200m 2Gi / 2Gi data:20Gi default clickhouse 200m / 200m 2Gi / 2Gi data:20Gi default Images: COMPONENT COMPONENT-DEFINITION IMAGE ch-keeper clickhouse-keeper-1.0.1 docker.io/apecloud/clickhouse:22.8.21-debian-11-r33 clickhouse clickhouse-1.0.1 docker.io/apecloud/clickhouse:22.8.21-debian-11-r33 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-keddj clkhouse-ccrhgq `kbcli cluster label clkhouse-ccrhgq app.kubernetes.io/instance- --namespace ns-keddj ` label "app.kubernetes.io/instance" not found. `kbcli cluster label clkhouse-ccrhgq app.kubernetes.io/instance=clkhouse-ccrhgq --namespace ns-keddj ` `kbcli cluster label clkhouse-ccrhgq --list --namespace ns-keddj ` NAME NAMESPACE LABELS clkhouse-ccrhgq ns-keddj app.kubernetes.io/instance=clkhouse-ccrhgq clusterdefinition.kubeblocks.io/name=clickhouse label cluster app.kubernetes.io/instance=clkhouse-ccrhgq Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=clkhouse-ccrhgq --namespace ns-keddj ` `kbcli cluster label clkhouse-ccrhgq --list --namespace ns-keddj ` NAME NAMESPACE LABELS clkhouse-ccrhgq ns-keddj app.kubernetes.io/instance=clkhouse-ccrhgq case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=clickhouse label cluster case.name=kbcli.test1 Success `kbcli cluster label clkhouse-ccrhgq case.name=kbcli.test2 --overwrite --namespace ns-keddj ` `kbcli cluster label clkhouse-ccrhgq --list --namespace ns-keddj ` NAME NAMESPACE LABELS clkhouse-ccrhgq ns-keddj app.kubernetes.io/instance=clkhouse-ccrhgq case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=clickhouse label cluster case.name=kbcli.test2 Success `kbcli cluster label clkhouse-ccrhgq case.name- --namespace ns-keddj ` `kbcli cluster label clkhouse-ccrhgq --list --namespace ns-keddj ` NAME NAMESPACE LABELS clkhouse-ccrhgq ns-keddj app.kubernetes.io/instance=clkhouse-ccrhgq clusterdefinition.kubeblocks.io/name=clickhouse delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT * FROM system.clusters"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash ` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file default 1 1 1 clkhouse-ccrhgq-clickhouse-4wc-0.clkhouse-ccrhgq-clickhouse-4wc-headless.ns-keddj.svc.cluster.local 10.244.4.120 9000 1 admin 0 0 0 default 1 1 2 clkhouse-ccrhgq-clickhouse-4wc-1.clkhouse-ccrhgq-clickhouse-4wc-headless.ns-keddj.svc.cluster.local 10.244.1.142 9000 0 admin 0 0 0 default 2 1 1 clkhouse-ccrhgq-clickhouse-c94-0.clkhouse-ccrhgq-clickhouse-c94-headless.ns-keddj.svc.cluster.local 10.244.4.92 9000 0 admin 0 0 0 default 2 1 2 clkhouse-ccrhgq-clickhouse-c94-1.clkhouse-ccrhgq-clickhouse-c94-headless.ns-keddj.svc.cluster.local 10.244.1.190 9000 0 admin 0 0 0 connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-clkhouse-ccrhgq" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-clkhouse-ccrhgq --namespace ns-keddj ` Error from server (NotFound): pods "test-db-client-executionloop-clkhouse-ccrhgq" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-clkhouse-ccrhgq" not found `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-clkhouse-ccrhgq namespace: ns-keddj spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local" - "--user" - "admin" - "--password" - "7M6Rq5c2x0" - "--port" - "8123" - "--dbtype" - "clickhouse" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" - "--cluster" - "default" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-clkhouse-ccrhgq.yaml` pod/test-db-client-executionloop-clkhouse-ccrhgq created apply test-db-client-executionloop-clkhouse-ccrhgq.yaml Success `rm -rf test-db-client-executionloop-clkhouse-ccrhgq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 31s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 36s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 52s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 1/1 Running 0 62s check pod test-db-client-executionloop-clkhouse-ccrhgq status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ccrhgq 0/1 Completed 0 68s check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:26 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:26 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:26 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:26 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --user admin --password 7M6Rq5c2x0 --port 8123 --dbtype clickhouse --test executionloop --duration 60 --interval 1 --cluster default SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 12:28:37.915 [main] WARN r.yandex.clickhouse.ClickHouseDriver - ****************************************************************************************** 12:28:37.917 [main] WARN r.yandex.clickhouse.ClickHouseDriver - * This driver is DEPRECATED. Please use [com.clickhouse.jdbc.ClickHouseDriver] instead. * 12:28:37.917 [main] WARN r.yandex.clickhouse.ClickHouseDriver - * Also everything in package [ru.yandex.clickhouse] will be removed starting from 0.4.0. * 12:28:37.917 [main] WARN r.yandex.clickhouse.ClickHouseDriver - ****************************************************************************************** Execution loop start: create database executions_loop CREATE DATABASE IF NOT EXISTS executions_loop ON CLUSTER default; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table ON CLUSTER default; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table ON CLUSTER default (id UInt32, value String) ENGINE = ReplicatedMergeTree() ORDER BY id; Execution loop start:INSERT INTO executions_loop.executions_loop_table (id, value) VALUES (1, 'executions_loop_test_1'); [ 1s ] executions total: 6 successful: 6 failed: 0 disconnect: 0 [ 2s ] executions total: 27 successful: 27 failed: 0 disconnect: 0 [ 3s ] executions total: 46 successful: 46 failed: 0 disconnect: 0 [ 4s ] executions total: 67 successful: 67 failed: 0 disconnect: 0 [ 5s ] executions total: 87 successful: 87 failed: 0 disconnect: 0 [ 6s ] executions total: 108 successful: 108 failed: 0 disconnect: 0 [ 7s ] executions total: 128 successful: 128 failed: 0 disconnect: 0 [ 8s ] executions total: 148 successful: 148 failed: 0 disconnect: 0 [ 9s ] executions total: 169 successful: 169 failed: 0 disconnect: 0 [ 10s ] executions total: 190 successful: 190 failed: 0 disconnect: 0 [ 11s ] executions total: 210 successful: 210 failed: 0 disconnect: 0 [ 12s ] executions total: 231 successful: 231 failed: 0 disconnect: 0 [ 13s ] executions total: 252 successful: 252 failed: 0 disconnect: 0 [ 14s ] executions total: 272 successful: 272 failed: 0 disconnect: 0 [ 15s ] executions total: 293 successful: 293 failed: 0 disconnect: 0 [ 16s ] executions total: 314 successful: 314 failed: 0 disconnect: 0 [ 17s ] executions total: 335 successful: 335 failed: 0 disconnect: 0 [ 18s ] executions total: 353 successful: 353 failed: 0 disconnect: 0 [ 19s ] executions total: 374 successful: 374 failed: 0 disconnect: 0 [ 20s ] executions total: 395 successful: 395 failed: 0 disconnect: 0 [ 21s ] executions total: 413 successful: 413 failed: 0 disconnect: 0 [ 22s ] executions total: 433 successful: 433 failed: 0 disconnect: 0 [ 23s ] executions total: 455 successful: 455 failed: 0 disconnect: 0 [ 24s ] executions total: 476 successful: 476 failed: 0 disconnect: 0 [ 25s ] executions total: 495 successful: 495 failed: 0 disconnect: 0 [ 26s ] executions total: 517 successful: 517 failed: 0 disconnect: 0 [ 27s ] executions total: 539 successful: 539 failed: 0 disconnect: 0 [ 28s ] executions total: 560 successful: 560 failed: 0 disconnect: 0 [ 29s ] executions total: 581 successful: 581 failed: 0 disconnect: 0 [ 30s ] executions total: 603 successful: 603 failed: 0 disconnect: 0 [ 31s ] executions total: 624 successful: 624 failed: 0 disconnect: 0 [ 32s ] executions total: 628 successful: 628 failed: 0 disconnect: 0 [ 33s ] executions total: 633 successful: 633 failed: 0 disconnect: 0 [ 34s ] executions total: 655 successful: 655 failed: 0 disconnect: 0 [ 35s ] executions total: 677 successful: 677 failed: 0 disconnect: 0 [ 36s ] executions total: 692 successful: 692 failed: 0 disconnect: 0 [ 37s ] executions total: 713 successful: 713 failed: 0 disconnect: 0 [ 38s ] executions total: 733 successful: 733 failed: 0 disconnect: 0 [ 39s ] executions total: 754 successful: 754 failed: 0 disconnect: 0 [ 40s ] executions total: 774 successful: 774 failed: 0 disconnect: 0 [ 41s ] executions total: 794 successful: 794 failed: 0 disconnect: 0 [ 42s ] executions total: 814 successful: 814 failed: 0 disconnect: 0 [ 43s ] executions total: 834 successful: 834 failed: 0 disconnect: 0 [ 44s ] executions total: 854 successful: 854 failed: 0 disconnect: 0 [ 45s ] executions total: 875 successful: 875 failed: 0 disconnect: 0 [ 46s ] executions total: 897 successful: 897 failed: 0 disconnect: 0 [ 47s ] executions total: 918 successful: 918 failed: 0 disconnect: 0 [ 48s ] executions total: 940 successful: 940 failed: 0 disconnect: 0 [ 49s ] executions total: 961 successful: 961 failed: 0 disconnect: 0 [ 50s ] executions total: 982 successful: 982 failed: 0 disconnect: 0 [ 51s ] executions total: 999 successful: 999 failed: 0 disconnect: 0 [ 52s ] executions total: 1018 successful: 1018 failed: 0 disconnect: 0 [ 53s ] executions total: 1039 successful: 1039 failed: 0 disconnect: 0 [ 54s ] executions total: 1059 successful: 1059 failed: 0 disconnect: 0 [ 60s ] executions total: 1079 successful: 1079 failed: 0 disconnect: 0 Test Result: Total Executions: 1079 Successful Executions: 1079 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: clickhouse Host: clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local Port: 8123 Database: Table: User: admin Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds Cluster: default DB_CLIENT_BATCH_DATA_COUNT: 1079 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-clkhouse-ccrhgq --namespace ns-keddj ` pod/test-db-client-executionloop-clkhouse-ccrhgq patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-clkhouse-ccrhgq" force deleted cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart clkhouse-ccrhgq --auto-approve --force=true --components clickhouse --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-restart-dnf8v created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-restart-dnf8v -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-restart-dnf8v ns-keddj Restart clkhouse-ccrhgq clickhouse Running 0/4 Sep 11,2025 20:29 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-restart-dnf8v ns-keddj Restart clkhouse-ccrhgq clickhouse Succeed 4/4 Sep 11,2025 20:29 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-restart-dnf8v ns-keddj Restart clkhouse-ccrhgq clickhouse Succeed 4/4 Sep 11,2025 20:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-restart-dnf8v --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-restart-dnf8v patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-restart-dnf8v --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-restart-dnf8v deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ccrhgq" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-clkhouse-ccrhgq.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-clkhouse-ccrhgq created apply test-chaos-mesh-dnserror-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-dnserror-clkhouse-ccrhgq.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ccrhgq" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ccrhgq" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-clkhouse-ccrhgq.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-clkhouse-ccrhgq created apply test-chaos-mesh-dnsrandom-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-dnsrandom-clkhouse-ccrhgq.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ccrhgq" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=clkhouse-ccrhgq,apps.kubeblocks.io/component-name=clickhouse,apps.kubeblocks.io/vct-name=data --namespace ns-keddj ` No resources found in ns-keddj namespace. clkhouse-ccrhgq clickhouse data pvc is empty cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ccrhgq namespace. `kbcli cluster volume-expand clkhouse-ccrhgq --auto-approve --force=true --components clickhouse --volume-claim-templates data --storage 22Gi --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-volumeexpansion-2gqqk created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-volumeexpansion-2gqqk -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-volumeexpansion-2gqqk ns-keddj VolumeExpansion clkhouse-ccrhgq clickhouse Running 0/4 Sep 11,2025 20:35 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 200m / 200m 2Gi / 2Gi data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:24 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done No resources found in clkhouse-ccrhgq namespace. check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-volumeexpansion-2gqqk ns-keddj VolumeExpansion clkhouse-ccrhgq clickhouse Succeed 4/4 Sep 11,2025 20:35 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-volumeexpansion-2gqqk ns-keddj VolumeExpansion clkhouse-ccrhgq clickhouse Succeed 4/4 Sep 11,2025 20:35 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-volumeexpansion-2gqqk --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-volumeexpansion-2gqqk patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-volumeexpansion-2gqqk --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-volumeexpansion-2gqqk deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success check component ch-keeper exists `kubectl get components -l app.kubernetes.io/instance=clkhouse-ccrhgq,apps.kubeblocks.io/component-name=ch-keeper --namespace ns-keddj | (grep "ch-keeper" || true )` check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale clkhouse-ccrhgq --auto-approve --force=true --components ch-keeper --cpu 300m --memory 2.1Gi --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-verticalscaling-sthp9 created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-verticalscaling-sthp9 -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-verticalscaling-sthp9 ns-keddj VerticalScaling clkhouse-ccrhgq ch-keeper Running 0/1 Sep 11,2025 20:40 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:20Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:40 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-verticalscaling-sthp9 ns-keddj VerticalScaling clkhouse-ccrhgq ch-keeper Succeed 1/1 Sep 11,2025 20:40 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-verticalscaling-sthp9 ns-keddj VerticalScaling clkhouse-ccrhgq ch-keeper Succeed 1/1 Sep 11,2025 20:40 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-verticalscaling-sthp9 --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-verticalscaling-sthp9 patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-verticalscaling-sthp9 --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-verticalscaling-sthp9 deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success check component ch-keeper exists `kubectl get components -l app.kubernetes.io/instance=clkhouse-ccrhgq,apps.kubeblocks.io/component-name=ch-keeper --namespace ns-keddj | (grep "ch-keeper" || true )` `kubectl get pvc -l app.kubernetes.io/instance=clkhouse-ccrhgq,apps.kubeblocks.io/component-name=ch-keeper,apps.kubeblocks.io/vct-name=data --namespace ns-keddj ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ccrhgq namespace. `kbcli cluster volume-expand clkhouse-ccrhgq --auto-approve --force=true --components ch-keeper --volume-claim-templates data --storage 21Gi --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-volumeexpansion-hh4jg created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-volumeexpansion-hh4jg -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-volumeexpansion-hh4jg ns-keddj VolumeExpansion clkhouse-ccrhgq ch-keeper Running 0/1 Sep 11,2025 20:41 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:40 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:30 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:29 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done No resources found in clkhouse-ccrhgq namespace. check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-volumeexpansion-hh4jg ns-keddj VolumeExpansion clkhouse-ccrhgq ch-keeper Succeed 1/1 Sep 11,2025 20:41 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-volumeexpansion-hh4jg ns-keddj VolumeExpansion clkhouse-ccrhgq ch-keeper Succeed 1/1 Sep 11,2025 20:41 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-volumeexpansion-hh4jg --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-volumeexpansion-hh4jg patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-volumeexpansion-hh4jg --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-volumeexpansion-hh4jg deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success No resources found in ns-keddj namespace. `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "CREATE TABLE test_kbcli (id Int32,name String) ENGINE = MergeTree() ORDER BY id;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` `echo "clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password \"7M6Rq5c2x0\" --query \"INSERT INTO test_kbcli VALUES (1,'pboiy');\" " | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Received exception from server (version 22.8.21): Code: 60. DB::Exception: Received from clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9000. DB::Exception: Table default.test_kbcli doesn't exist. (UNKNOWN_TABLE) (query: INSERT INTO test_kbcli VALUES (1,'pboiy');) command terminated with exit code 60 `echo "clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password \"7M6Rq5c2x0\" --query \"INSERT INTO test_kbcli VALUES (1,'pboiy');\" " | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Received exception from server (version 22.8.21): Code: 57. DB::Exception: Received from clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9000. DB::Exception: Table default.test_kbcli already exists. (TABLE_ALREADY_EXISTS) (query: CREATE TABLE test_kbcli (id Int32,name String) ENGINE = MergeTree() ORDER BY id;) command terminated with exit code 57 `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "CREATE TABLE test_kbcli (id Int32,name String) ENGINE = MergeTree() ORDER BY id;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Received exception from server (version 22.8.21): Code: 60. DB::Exception: Received from clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local:9000. DB::Exception: Table default.test_kbcli doesn't exist. (UNKNOWN_TABLE) (query: INSERT INTO test_kbcli VALUES (1,'pboiy');) command terminated with exit code 60 `echo "clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password \"7M6Rq5c2x0\" --query \"INSERT INTO test_kbcli VALUES (1,'pboiy');\" " | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file `clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT * FROM test_kbcli;"` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg: Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg: Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg: Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg:1 pboiy check msg:[pboiy] equal msg:[1 pboiy] cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart clkhouse-ccrhgq --auto-approve --force=true --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-restart-sfjsk created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-restart-sfjsk -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-restart-sfjsk ns-keddj Restart clkhouse-ccrhgq ch-keeper,clickhouse Running 0/5 Sep 11,2025 20:43 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-restart-sfjsk ns-keddj Restart clkhouse-ccrhgq ch-keeper,clickhouse Succeed 5/5 Sep 11,2025 20:43 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-restart-sfjsk ns-keddj Restart clkhouse-ccrhgq ch-keeper,clickhouse Succeed 5/5 Sep 11,2025 20:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-restart-sfjsk --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-restart-sfjsk patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-restart-sfjsk --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-restart-sfjsk deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover fullcpu check cluster status before cluster-failover-fullcpu check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ccrhgq" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-clkhouse-ccrhgq.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-clkhouse-ccrhgq created apply test-chaos-mesh-fullcpu-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-fullcpu-clkhouse-ccrhgq.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ccrhgq" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq created apply test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ccrhgq" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-clkhouse-ccrhgq.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-clkhouse-ccrhgq created apply test-chaos-mesh-networkdelay-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-networkdelay-clkhouse-ccrhgq.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ccrhgq" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running `kill 1` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq created apply test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ccrhgq" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 action: partition mode: all target: mode: all selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-clkhouse-ccrhgq.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-clkhouse-ccrhgq created apply test-chaos-mesh-networkpartition-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-networkpartition-clkhouse-ccrhgq.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ccrhgq" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ccrhgq" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-clkhouse-ccrhgq.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-clkhouse-ccrhgq created apply test-chaos-mesh-networkduplicate-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-networkduplicate-clkhouse-ccrhgq.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ccrhgq" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-clkhouse-ccrhgq" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-clkhouse-ccrhgq.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-clkhouse-ccrhgq created apply test-chaos-mesh-oom-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-oom-clkhouse-ccrhgq.yaml` check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-clkhouse-ccrhgq" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-clkhouse-ccrhgq patched check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ccrhgq" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-clkhouse-ccrhgq.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-clkhouse-ccrhgq created apply test-chaos-mesh-podfailure-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-podfailure-clkhouse-ccrhgq.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Abnormal Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ccrhgq" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success cluster clickhouse scale-out cluster clickhouse scale-out replicas: 3 check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ccrhgq namespace. `kbcli cluster scale-out clkhouse-ccrhgq --auto-approve --force=true --components clickhouse --replicas 1 --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-horizontalscaling-qtq7r created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-horizontalscaling-qtq7r -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-horizontalscaling-qtq7r ns-keddj HorizontalScaling clkhouse-ccrhgq clickhouse Running 0/2 Sep 11,2025 21:07 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-2 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:07 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-2 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:07 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done No resources found in clkhouse-ccrhgq namespace. check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-horizontalscaling-qtq7r ns-keddj HorizontalScaling clkhouse-ccrhgq clickhouse Succeed 2/2 Sep 11,2025 21:07 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-horizontalscaling-qtq7r ns-keddj HorizontalScaling clkhouse-ccrhgq clickhouse Succeed 2/2 Sep 11,2025 21:07 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-horizontalscaling-qtq7r --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-horizontalscaling-qtq7r patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-horizontalscaling-qtq7r --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-horizontalscaling-qtq7r deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success cluster clickhouse scale-in cluster clickhouse scale-in replicas: 2 check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ccrhgq namespace. `kbcli cluster scale-in clkhouse-ccrhgq --auto-approve --force=true --components clickhouse --replicas 1 --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-horizontalscaling-hkzhp created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-horizontalscaling-hkzhp -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-horizontalscaling-hkzhp ns-keddj HorizontalScaling clkhouse-ccrhgq clickhouse Running 0/2 Sep 11,2025 21:08 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done No resources found in clkhouse-ccrhgq namespace. check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-horizontalscaling-hkzhp ns-keddj HorizontalScaling clkhouse-ccrhgq clickhouse Succeed 2/2 Sep 11,2025 21:08 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-horizontalscaling-hkzhp ns-keddj HorizontalScaling clkhouse-ccrhgq clickhouse Succeed 2/2 Sep 11,2025 21:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-horizontalscaling-hkzhp --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-horizontalscaling-hkzhp patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-horizontalscaling-hkzhp --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-horizontalscaling-hkzhp deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ccrhgq" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-clkhouse-ccrhgq.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-clkhouse-ccrhgq created apply test-chaos-mesh-networklossover-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-networklossover-clkhouse-ccrhgq.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:48 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 20:44 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000001/10.224.0.5 Sep 11,2025 20:45 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 20:44 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ccrhgq" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop clkhouse-ccrhgq --auto-approve --force=true --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-stop-gxwsp created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-stop-gxwsp -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-stop-gxwsp ns-keddj Stop clkhouse-ccrhgq Running -/- Sep 11,2025 21:11 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Stopped Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-stop-gxwsp ns-keddj Stop clkhouse-ccrhgq ch-keeper,clickhouse Succeed 5/5 Sep 11,2025 21:11 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-stop-gxwsp ns-keddj Stop clkhouse-ccrhgq ch-keeper,clickhouse Succeed 5/5 Sep 11,2025 21:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-stop-gxwsp --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-stop-gxwsp patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-stop-gxwsp --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-stop-gxwsp deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start clkhouse-ccrhgq --force=true --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-start-gx4c6 created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-start-gx4c6 -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-start-gx4c6 ns-keddj Start clkhouse-ccrhgq ch-keeper,clickhouse Running 0/5 Sep 11,2025 21:11 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 200m / 200m 2Gi / 2Gi data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:11 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-start-gx4c6 ns-keddj Start clkhouse-ccrhgq ch-keeper,clickhouse Succeed 5/5 Sep 11,2025 21:11 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-start-gx4c6 ns-keddj Start clkhouse-ccrhgq ch-keeper,clickhouse Succeed 5/5 Sep 11,2025 21:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-start-gx4c6 --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-start-gx4c6 patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-start-gx4c6 --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-start-gx4c6 deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale clkhouse-ccrhgq --auto-approve --force=true --components clickhouse --cpu 300m --memory 2.1Gi --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-verticalscaling-gzkxz created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ccrhgq-verticalscaling-gzkxz -n ns-keddj check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-verticalscaling-gzkxz ns-keddj VerticalScaling clkhouse-ccrhgq clickhouse Running 0/4 Sep 11,2025 21:13 UTC+0800 check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ccrhgq --status all --namespace ns-keddj ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ccrhgq-verticalscaling-gzkxz ns-keddj VerticalScaling clkhouse-ccrhgq clickhouse Succeed 4/4 Sep 11,2025 21:13 UTC+0800 check ops status done ops_status:clkhouse-ccrhgq-verticalscaling-gzkxz ns-keddj VerticalScaling clkhouse-ccrhgq clickhouse Succeed 4/4 Sep 11,2025 21:13 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ccrhgq-verticalscaling-gzkxz --namespace ns-keddj ` opsrequest.operations.kubeblocks.io/clkhouse-ccrhgq-verticalscaling-gzkxz patched `kbcli cluster delete-ops --name clkhouse-ccrhgq-verticalscaling-gzkxz --force --auto-approve --namespace ns-keddj ` OpsRequest clkhouse-ccrhgq-verticalscaling-gzkxz deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ccrhgq" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-clkhouse-ccrhgq.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-clkhouse-ccrhgq created apply test-chaos-mesh-timeoffset-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-timeoffset-clkhouse-ccrhgq.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ccrhgq" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ccrhgq" not found check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running Error from server (NotFound): pods "test-db-client-connectionstress-clkhouse-ccrhgq" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-clkhouse-ccrhgq --namespace ns-keddj ` Error from server (NotFound): pods "test-db-client-connectionstress-clkhouse-ccrhgq" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-clkhouse-ccrhgq" not found `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-clkhouse-ccrhgq namespace: ns-keddj spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local" - "--user" - "admin" - "--password" - "7M6Rq5c2x0" - "--port" - "8123" - "--database" - "default" - "--dbtype" - "clickhouse" - "--test" - "connectionstress" - "--connections" - "4096" - "--duration" - "60" - "--cluster" - "default" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-clkhouse-ccrhgq.yaml` pod/test-db-client-connectionstress-clkhouse-ccrhgq created apply test-db-client-connectionstress-clkhouse-ccrhgq.yaml Success `rm -rf test-db-client-connectionstress-clkhouse-ccrhgq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ccrhgq 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ccrhgq 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ccrhgq 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ccrhgq 1/1 Running 0 20s check pod test-db-client-connectionstress-clkhouse-ccrhgq status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ccrhgq 0/1 Completed 0 26s check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --user admin --password 7M6Rq5c2x0 --port 8123 --database default --dbtype clickhouse --test connectionstress --connections 4096 --duration 60 --cluster default SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 13:16:47.945 [main] WARN r.yandex.clickhouse.ClickHouseDriver - ****************************************************************************************** 13:16:47.947 [main] WARN r.yandex.clickhouse.ClickHouseDriver - * This driver is DEPRECATED. Please use [com.clickhouse.jdbc.ClickHouseDriver] instead. * 13:16:47.947 [main] WARN r.yandex.clickhouse.ClickHouseDriver - * Also everything in package [ru.yandex.clickhouse] will be removed starting from 0.4.0. * 13:16:47.947 [main] WARN r.yandex.clickhouse.ClickHouseDriver - ****************************************************************************************** Test Result: null Connection Information: Database Type: clickhouse Host: clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local Port: 8123 Database: default Table: User: admin Org: Access Mode: mysql Test Type: connectionstress Connection Count: 4096 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-clkhouse-ccrhgq --namespace ns-keddj ` pod/test-db-client-connectionstress-clkhouse-ccrhgq patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-clkhouse-ccrhgq" force deleted check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-clkhouse-ccrhgq" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-clkhouse-ccrhgq" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-clkhouse-ccrhgq namespace: ns-keddj spec: selector: namespaces: - ns-keddj labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ccrhgq-clickhouse-4wc-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-clkhouse-ccrhgq.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-clkhouse-ccrhgq created apply test-chaos-mesh-podkill-clkhouse-ccrhgq.yaml Success `rm -rf test-chaos-mesh-podkill-clkhouse-ccrhgq.yaml` check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse DoNotTerminate Updating Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:17 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-clkhouse-ccrhgq --namespace ns-keddj ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-clkhouse-ccrhgq" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-clkhouse-ccrhgq patched check failover pod name failover pod name:clkhouse-ccrhgq-clickhouse-4wc-0 failover podkill Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check db_client batch data count `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update clkhouse-ccrhgq --termination-policy=WipeOut --namespace ns-keddj ` cluster.apps.kubeblocks.io/clkhouse-ccrhgq updated check cluster status `kbcli cluster list clkhouse-ccrhgq --show-labels --namespace ns-keddj ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ccrhgq ns-keddj clickhouse WipeOut Running Sep 11,2025 20:24 UTC+0800 app.kubernetes.io/instance=clkhouse-ccrhgq,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ccrhgq --namespace ns-keddj ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ccrhgq-ch-keeper-0 ns-keddj clkhouse-ccrhgq ch-keeper Running leader 0 300m / 300m 2254857830400m / 2254857830400m data:21Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:11 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:17 UTC+0800 clkhouse-ccrhgq-clickhouse-4wc-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-4wc) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-0 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000002/10.224.0.7 Sep 11,2025 21:13 UTC+0800 clkhouse-ccrhgq-clickhouse-c94-1 ns-keddj clkhouse-ccrhgq clickhouse(clickhouse-c94) Running 0 300m / 300m 2254857830400m / 2254857830400m data:22Gi aks-cicdamdpool-42425418-vmss000000/10.224.0.9 Sep 11,2025 21:13 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ccrhgq` set secret: clkhouse-ccrhgq-clickhouse-c94-account-admin `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ccrhgq-clickhouse-c94-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:7M6Rq5c2x0;DB_PORT:9000;DB_DATABASE:default No resources found in ns-keddj namespace. check cluster connect `echo 'clickhouse-client --host clkhouse-ccrhgq-clickhouse-4wc.ns-keddj.svc.cluster.local --port 9000 --user admin --password "7M6Rq5c2x0"' | kubectl exec -it clkhouse-ccrhgq-clickhouse-4wc-0 --namespace ns-keddj -- bash` check cluster connect done cluster list-logs `kbcli cluster list-logs clkhouse-ccrhgq --namespace ns-keddj ` No log files found. Error from server (NotFound): pods "clkhouse-ccrhgq-clickhouse-4wc-0" not found cluster logs `kbcli cluster logs clkhouse-ccrhgq --tail 30 --namespace ns-keddj ` Defaulted container "clickhouse" out of: clickhouse, kbagent, copy-tools (init), init-kbagent (init), kbagent-worker (init) 0. DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0xa40325a in /opt/bitnami/clickhouse/bin/clickhouse 1. DB::throwFromErrnoWithPath(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, int, int) @ 0xa4042a0 in /opt/bitnami/clickhouse/bin/clickhouse 2. DB::ReadBufferFromFileDescriptor::nextImpl() @ 0xa46390e in /opt/bitnami/clickhouse/bin/clickhouse 3. DB::AsynchronousMetrics::BlockDeviceStatValues::read(DB::ReadBuffer&) @ 0x146e2fd1 in /opt/bitnami/clickhouse/bin/clickhouse 4. DB::AsynchronousMetrics::update(std::__1::chrono::time_point > >) @ 0x146d7d6f in /opt/bitnami/clickhouse/bin/clickhouse 5. DB::AsynchronousMetrics::run() @ 0x146e24be in /opt/bitnami/clickhouse/bin/clickhouse 6. ? @ 0x146e6bac in /opt/bitnami/clickhouse/bin/clickhouse 7. ThreadPoolImpl::worker(std::__1::__list_iterator) @ 0xa4c4828 in /opt/bitnami/clickhouse/bin/clickhouse 8. ? @ 0xa4c7a3d in /opt/bitnami/clickhouse/bin/clickhouse 9. start_thread @ 0x7ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so 10. clone @ 0xfba2f in /lib/x86_64-linux-gnu/libc-2.31.so (version 22.8.21.38 (official build)) 2025.09.11 13:17:30.117212 [ 275 ] *** KeeperTCPHandler: Requesting session ID for the new client 2025.09.11 13:17:30.123769 [ 275 ] *** KeeperTCPHandler: Received session ID 35 2025.09.11 13:17:55.621674 [ 75 ] *** KeeperDispatcher: Found dead session 34, will try to close it 2025.09.11 13:17:55.621765 [ 75 ] *** KeeperDispatcher: Dead session close request pushed 2025.09.11 13:18:27.000665 [ 148 ] *** void DB::AsynchronousMetrics::update(std::chrono::system_clock::time_point): Code: 74. DB::ErrnoException: Cannot read from file: /sys/block/sdp/stat, errno: 19, strerror: No such device. (CANNOT_READ_FROM_FILE_DESCRIPTOR), Stack trace (when copying this message, always include the lines below): 0. DB::Exception::Exception(std::__1::basic_string, std::__1::allocator > const&, int, bool) @ 0xa40325a in /opt/bitnami/clickhouse/bin/clickhouse 1. DB::throwFromErrnoWithPath(std::__1::basic_string, std::__1::allocator > const&, std::__1::basic_string, std::__1::allocator > const&, int, int) @ 0xa4042a0 in /opt/bitnami/clickhouse/bin/clickhouse 2. DB::ReadBufferFromFileDescriptor::nextImpl() @ 0xa46390e in /opt/bitnami/clickhouse/bin/clickhouse 3. DB::AsynchronousMetrics::BlockDeviceStatValues::read(DB::ReadBuffer&) @ 0x146e2fd1 in /opt/bitnami/clickhouse/bin/clickhouse 4. DB::AsynchronousMetrics::update(std::__1::chrono::time_point > >) @ 0x146d7d6f in /opt/bitnami/clickhouse/bin/clickhouse 5. DB::AsynchronousMetrics::run() @ 0x146e24be in /opt/bitnami/clickhouse/bin/clickhouse 6. ? @ 0x146e6bac in /opt/bitnami/clickhouse/bin/clickhouse 7. ThreadPoolImpl::worker(std::__1::__list_iterator) @ 0xa4c4828 in /opt/bitnami/clickhouse/bin/clickhouse 8. ? @ 0xa4c7a3d in /opt/bitnami/clickhouse/bin/clickhouse 9. start_thread @ 0x7ea7 in /lib/x86_64-linux-gnu/libpthread-2.31.so 10. clone @ 0xfba2f in /lib/x86_64-linux-gnu/libc-2.31.so (version 22.8.21.38 (official build)) delete cluster clkhouse-ccrhgq `kbcli cluster delete clkhouse-ccrhgq --auto-approve --namespace ns-keddj ` Cluster clkhouse-ccrhgq deleted pod_info:clkhouse-ccrhgq-ch-keeper-0 2/2 Running 0 6m54s clkhouse-ccrhgq-clickhouse-4wc-0 2/2 Running 0 70s clkhouse-ccrhgq-clickhouse-4wc-1 2/2 Running 0 5m15s clkhouse-ccrhgq-clickhouse-c94-0 2/2 Running 0 4m49s clkhouse-ccrhgq-clickhouse-c94-1 2/2 Running 0 5m15s No resources found in ns-keddj namespace. delete cluster pod done No resources found in ns-keddj namespace. check cluster resource non-exist OK: pvc No resources found in ns-keddj namespace. delete cluster done No resources found in ns-keddj namespace. No resources found in ns-keddj namespace. No resources found in ns-keddj namespace. Clickhouse Test Suite All Done! Test Engine: clickhouse Test Type: 29 --------------------------------------Clickhouse (Topology = cluster Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=clickhouse-1.0.1;ComponentVersion=clickhouse;ServiceVersion=22.8.21;]|[Description=Create a cluster with the specified component definition clickhouse-1.0.1 and component version clickhouse and service version 22.8.21] [PASSED]|[Connect]|[ComponentName=clickhouse]|[Description=Connect to the cluster] [PASSED]|[Restart]|[ComponentName=clickhouse]|[Description=Restart the cluster specify component clickhouse] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[VolumeExpansion]|[ComponentName=clickhouse]|[Description=VolumeExpansion the cluster specify component clickhouse] [PASSED]|[VerticalScaling]|[ComponentName=ch-keeper]|[Description=VerticalScaling the cluster specify component ch-keeper] [PASSED]|[VolumeExpansion]|[ComponentName=ch-keeper]|[Description=VolumeExpansion the cluster specify component ch-keeper] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[No-Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[No-Failover]|[HA=Kill 1;ComponentName=clickhouse]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[No-Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[No-Failover]|[HA=OOM;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[No-Failover]|[HA=Pod Failure;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[HorizontalScaling Out]|[ComponentName=clickhouse]|[Description=HorizontalScaling Out the cluster specify component clickhouse] [PASSED]|[HorizontalScaling In]|[ComponentName=clickhouse]|[Description=HorizontalScaling In the cluster specify component clickhouse] [PASSED]|[No-Failover]|[HA=Network Loss;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[VerticalScaling]|[ComponentName=clickhouse]|[Description=VerticalScaling the cluster specify component clickhouse] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=clickhouse]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[No-Failover]|[HA=Pod Kill;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]