source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-bcrcc ` `kubectl create namespace ns-bcrcc` namespace/ns-bcrcc created create namespace ns-bcrcc done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.0` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 143M 0 --:--:-- --:--:-- --:--:-- 143M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.0 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Kubernetes Env: v1.32.5-eks-5d4a308 check snapshot controller check snapshot controller done eks default-vsc found POD_RESOURCES: No resources found found default storage class: gp3 KubeBlocks version is:1.0.0 skip upgrade KubeBlocks current KubeBlocks version: 1.0.0 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:clickhouse set component version set component version:clickhouse set service versions:24.8.3,22.9.4 set service versions sorted:22.9.4,24.8.3 set clickhouse component definition set clickhouse component definition clickhouse-24-1.0.0-alpha.0 set replicas first:2,22.9.4|2,24.8.3 set replicas third:2,24.8.3 set replicas fourth:2,24.8.3 set minimum cmpv service version set minimum cmpv service version replicas:2,24.8.3 REPORT_COUNT:1 CLUSTER_TOPOLOGY:cluster topology cluster found in cluster definition clickhouse set clickhouse component definition set clickhouse component definition clickhouse-keeper-24-1.0.0-alpha.0 LIMIT_CPU:0.2 LIMIT_MEMORY:1 storage size: 20 No resources found in ns-bcrcc namespace. termination_policy:WipeOut create 2 replica WipeOut clickhouse cluster check component definition set component definition by component version check cmpd by labels set component definition1: clickhouse-24-1.0.0-alpha.0 by component version:clickhouse apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: clkhouse-ppqueu namespace: ns-bcrcc spec: clusterDef: clickhouse topology: cluster terminationPolicy: WipeOut componentSpecs: - name: clickhouse serviceVersion: 24.8.3 replicas: 2 resources: requests: cpu: 200m memory: 1Gi limits: cpu: 200m memory: 1Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: ch-keeper serviceVersion: 24.8.3 replicas: 1 resources: requests: cpu: 200m memory: 1Gi limits: cpu: 200m memory: 1Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi `kubectl apply -f test_create_clkhouse-ppqueu.yaml` cluster.apps.kubeblocks.io/clkhouse-ppqueu created apply test_create_clkhouse-ppqueu.yaml Success `rm -rf test_create_clkhouse-ppqueu.yaml` check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut May 28,2025 11:39 UTC+0800 clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Creating cluster_status:Creating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check pod clkhouse-ppqueu-clickhouse-0 container_name clickhouse exist password ftiBthj7W37lwgZ0 check pod clkhouse-ppqueu-clickhouse-0 container_name kbagent exist password ftiBthj7W37lwgZ0 No container logs contain secret password. describe cluster `kbcli cluster describe clkhouse-ppqueu --namespace ns-bcrcc ` Name: clkhouse-ppqueu Created Time: May 28,2025 11:39 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-bcrcc clickhouse cluster Running WipeOut Endpoints: COMPONENT INTERNAL EXTERNAL clickhouse clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8001 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8443 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:9000 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:9004 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:9005 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:9009 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:9010 clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:9440 ch-keeper clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:8123 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:8443 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9000 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9009 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9010 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:8001 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9181 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9234 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9281 clkhouse-ppqueu-ch-keeper.ns-bcrcc.svc.cluster.local:9440 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME ch-keeper 24.8.3 clkhouse-ppqueu-ch-keeper-0 leader Running us-west-2a ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:39 UTC+0800 clickhouse 24.8.3 clkhouse-ppqueu-clickhouse-0 Running us-west-2a ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clickhouse 24.8.3 clkhouse-ppqueu-clickhouse-1 Running us-west-2a ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS clickhouse 200m / 200m 1Gi / 1Gi data:20Gi kb-default-sc ch-keeper 200m / 200m 1Gi / 1Gi data:20Gi kb-default-sc Images: COMPONENT COMPONENT-DEFINITION IMAGE clickhouse clickhouse-24-1.0.0-alpha.0 docker.io/apecloud/clickhouse:24.8.3-debian-12-r1 ch-keeper clickhouse-keeper-24-1.0.0-alpha.0 docker.io/apecloud/clickhouse:24.8.3-debian-12-r1 docker.io/apecloud/busybox:1.36 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-bcrcc clkhouse-ppqueu `kbcli cluster label clkhouse-ppqueu app.kubernetes.io/instance- --namespace ns-bcrcc ` label "app.kubernetes.io/instance" not found. `kbcli cluster label clkhouse-ppqueu app.kubernetes.io/instance=clkhouse-ppqueu --namespace ns-bcrcc ` `kbcli cluster label clkhouse-ppqueu --list --namespace ns-bcrcc ` NAME NAMESPACE LABELS clkhouse-ppqueu ns-bcrcc app.kubernetes.io/instance=clkhouse-ppqueu clusterdefinition.kubeblocks.io/name=clickhouse label cluster app.kubernetes.io/instance=clkhouse-ppqueu Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=clkhouse-ppqueu --namespace ns-bcrcc ` `kbcli cluster label clkhouse-ppqueu --list --namespace ns-bcrcc ` NAME NAMESPACE LABELS clkhouse-ppqueu ns-bcrcc app.kubernetes.io/instance=clkhouse-ppqueu case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=clickhouse label cluster case.name=kbcli.test1 Success `kbcli cluster label clkhouse-ppqueu case.name=kbcli.test2 --overwrite --namespace ns-bcrcc ` `kbcli cluster label clkhouse-ppqueu --list --namespace ns-bcrcc ` NAME NAMESPACE LABELS clkhouse-ppqueu ns-bcrcc app.kubernetes.io/instance=clkhouse-ppqueu case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=clickhouse label cluster case.name=kbcli.test2 Success `kbcli cluster label clkhouse-ppqueu case.name- --namespace ns-bcrcc ` `kbcli cluster label clkhouse-ppqueu --list --namespace ns-bcrcc ` NAME NAMESPACE LABELS clkhouse-ppqueu ns-bcrcc app.kubernetes.io/instance=clkhouse-ppqueu clusterdefinition.kubeblocks.io/name=clickhouse delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT * FROM system.clusters"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash ` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file default 1 1 1 1 clkhouse-ppqueu-clickhouse-0.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local 172.31.15.91 9000 1 admin 0 0 0 \N \N \N default 1 1 1 2 clkhouse-ppqueu-clickhouse-1.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local 172.31.9.16 9000 0 admin 0 0 0 \N \N \N connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-clkhouse-ppqueu" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-clkhouse-ppqueu --namespace ns-bcrcc ` Error from server (NotFound): pods "test-db-client-executionloop-clkhouse-ppqueu" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-clkhouse-ppqueu" not found `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-clkhouse-ppqueu namespace: ns-bcrcc spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local" - "--user" - "admin" - "--password" - "ftiBthj7W37lwgZ0" - "--port" - "8123" - "--dbtype" - "clickhouse" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-clkhouse-ppqueu.yaml` pod/test-db-client-executionloop-clkhouse-ppqueu created apply test-db-client-executionloop-clkhouse-ppqueu.yaml Success `rm -rf test-db-client-executionloop-clkhouse-ppqueu.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 59s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 65s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 1/1 Running 0 71s check pod test-db-client-executionloop-clkhouse-ppqueu status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-clkhouse-ppqueu 0/1 Completed 0 77s check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Transfer-Encoding: chunked[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Query-Id: 876bf7dc-0fa3-4f21-b068-7bce2b9d07bd[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Timezone: Etc/UTC[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Keep-Alive: timeout=10, max=9999[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"37","written_rows":"1","written_bytes":"37","total_rows_to_read":"0","result_rows":"1","result_bytes":"37","elapsed_ns":"18883215"***[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << HTTP/1.1 200 OK 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Date: Wed, 28 May 2025 03:43:40 GMT 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Connection: Keep-Alive 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Content-Type: text/plain; charset=UTF-8 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Server-Display-Name: clkhouse-ppqueu-clickhouse-0.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Transfer-Encoding: chunked 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Query-Id: 876bf7dc-0fa3-4f21-b068-7bce2b9d07bd 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Timezone: Etc/UTC 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Keep-Alive: timeout=10, max=9999 03:43:40.363 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"37","written_rows":"1","written_bytes":"37","total_rows_to_read":"0","result_rows":"1","result_bytes":"37","elapsed_ns":"18883215"*** 03:43:40.363 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Connection can be kept alive for 10000 MILLISECONDS 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "0[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "[\r][\n]" 03:43:40.363 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection [id: 1][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123] can be kept alive for 10.0 seconds 03:43:40.363 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-1: set socket timeout to 0 03:43:40.363 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection released: [id: 1][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 1; route allocated: 1 of 500; total allocated: 1 of 10000] 03:43:40.364 [main] DEBUG ru.yandex.clickhouse.ClickHouseStatementImpl -- Executing SQL: INSERT INTO executions_loop.executions_loop_table (id, value) VALUES (912, 'executions_loop_test_912') 03:43:40.364 [main] DEBUG ru.yandex.clickhouse.ClickHouseStatementImpl -- Request url: http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123/?compress=1&query_id=71809b11-522c-4bf9-aa6d-47f6ccb92671&extremes=0&database=default 03:43:40.364 [main] DEBUG org.apache.http.client.protocol.RequestAddCookies -- CookieSpec selected: standard 03:43:40.364 [main] DEBUG org.apache.http.client.protocol.RequestAuthCache -- Re-using cached 'basic' auth scheme for http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123 03:43:40.364 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection request: [route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 1; route allocated: 1 of 500; total allocated: 1 of 10000] 03:43:40.364 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection leased: [id: 1][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 0; route allocated: 1 of 500; total allocated: 1 of 10000] 03:43:40.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-1: set socket timeout to 0 03:43:40.364 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-1: set socket timeout to 30000 03:43:40.364 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Executing request POST /?compress=1&query_id=71809b11-522c-4bf9-aa6d-47f6ccb92671&extremes=0&database=default HTTP/1.1 03:43:40.364 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Target auth state: UNCHALLENGED 03:43:40.364 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Proxy auth state: UNCHALLENGED 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> POST /?compress=1&query_id=71809b11-522c-4bf9-aa6d-47f6ccb92671&extremes=0&database=default HTTP/1.1 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> Content-Length: 102 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> Content-Type: text/plain; charset=UTF-8 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> Host: clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> Connection: Keep-Alive 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> User-Agent: Apache-HttpClient/4.5.14 (Java/11.0.16) 03:43:40.365 [main] DEBUG org.apache.http.headers -- http-outgoing-1 >> Authorization: Basic YWRtaW46ZnRpQnRoajdXMzdsd2daMA== 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "POST /?compress=1&query_id=71809b11-522c-4bf9-aa6d-47f6ccb92671&extremes=0&database=default HTTP/1.1[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "Content-Length: 102[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "Host: clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "Connection: Keep-Alive[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/11.0.16)[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "Authorization: Basic YWRtaW46ZnRpQnRoajdXMzdsd2daMA==[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "[\r][\n]" 03:43:40.365 [main] DEBUG org.apache.http.wire -- http-outgoing-1 >> "INSERT INTO executions_loop.executions_loop_table (id, value) VALUES (912, 'executions_loop_test_912')" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "HTTP/1.1 200 OK[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Date: Wed, 28 May 2025 03:43:40 GMT[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Connection: Keep-Alive[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Content-Type: text/plain; charset=UTF-8[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Server-Display-Name: clkhouse-ppqueu-clickhouse-0.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Transfer-Encoding: chunked[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Query-Id: 71809b11-522c-4bf9-aa6d-47f6ccb92671[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Timezone: Etc/UTC[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "Keep-Alive: timeout=10, max=9998[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"37","written_rows":"1","written_bytes":"37","total_rows_to_read":"0","result_rows":"1","result_bytes":"37","elapsed_ns":"64380350"***[\r][\n]" 03:43:40.439 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "[\r][\n]" 03:43:40.440 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "0[\r][\n]" 03:43:40.440 [main] DEBUG org.apache.http.wire -- http-outgoing-1 << "[\r][\n]" 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << HTTP/1.1 200 OK 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Date: Wed, 28 May 2025 03:43:40 GMT 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Connection: Keep-Alive 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Content-Type: text/plain; charset=UTF-8 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Server-Display-Name: clkhouse-ppqueu-clickhouse-0.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Transfer-Encoding: chunked 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Query-Id: 71809b11-522c-4bf9-aa6d-47f6ccb92671 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Timezone: Etc/UTC 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << Keep-Alive: timeout=10, max=9998 03:43:40.440 [main] DEBUG org.apache.http.headers -- http-outgoing-1 << X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"37","written_rows":"1","written_bytes":"37","total_rows_to_read":"0","result_rows":"1","result_bytes":"37","elapsed_ns":"64380350"*** 03:43:40.440 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Connection can be kept alive for 10000 MILLISECONDS 03:43:40.440 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection [id: 1][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123] can be kept alive for 10.0 seconds 03:43:40.440 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-1: set socket timeout to 0 03:43:40.440 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection released: [id: 1][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 1; route allocated: 1 of 500; total allocated: 1 of 10000] [ 60s ] executions total: 912 successful: 912 failed: 0 disconnect: 0 03:43:40.442 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection manager is shutting down 03:43:40.442 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-1: Close connection 03:43:40.442 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection manager shut down Test Result: Total Executions: 912 Successful Executions: 912 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: clickhouse Host: clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local Port: 8123 Database: Table: User: admin Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds Cluster: DB_CLIENT_BATCH_DATA_COUNT: 912 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-clkhouse-ppqueu --namespace ns-bcrcc ` pod/test-db-client-executionloop-clkhouse-ppqueu patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-clkhouse-ppqueu" force deleted test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ppqueu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-clkhouse-ppqueu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-clkhouse-ppqueu created apply test-chaos-mesh-networkdelay-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-networkdelay-clkhouse-ppqueu.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ppqueu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success check component ch-keeper exists `kubectl get components -l app.kubernetes.io/instance=clkhouse-ppqueu,apps.kubeblocks.io/component-name=ch-keeper --namespace ns-bcrcc | (grep "ch-keeper" || true )` check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale clkhouse-ppqueu --auto-approve --force=true --components ch-keeper --cpu 300m --memory 1.1Gi --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-verticalscaling-4chjn created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-verticalscaling-4chjn -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-verticalscaling-4chjn ns-bcrcc VerticalScaling clkhouse-ppqueu ch-keeper Running 0/1 May 28,2025 11:46 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-verticalscaling-4chjn ns-bcrcc VerticalScaling clkhouse-ppqueu ch-keeper Succeed 1/1 May 28,2025 11:46 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-verticalscaling-4chjn ns-bcrcc VerticalScaling clkhouse-ppqueu ch-keeper Succeed 1/1 May 28,2025 11:46 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-verticalscaling-4chjn --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-verticalscaling-4chjn patched `kbcli cluster delete-ops --name clkhouse-ppqueu-verticalscaling-4chjn --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-verticalscaling-4chjn deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover kill1 Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-clkhouse-ppqueu" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-clkhouse-ppqueu --namespace ns-bcrcc ` Error from server (NotFound): pods "test-db-client-connectionstress-clkhouse-ppqueu" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-clkhouse-ppqueu" not found `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-clkhouse-ppqueu namespace: ns-bcrcc spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local" - "--user" - "admin" - "--password" - "ftiBthj7W37lwgZ0" - "--port" - "8123" - "--database" - "default" - "--dbtype" - "clickhouse" - "--test" - "connectionstress" - "--connections" - "4096" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-clkhouse-ppqueu.yaml` pod/test-db-client-connectionstress-clkhouse-ppqueu created apply test-db-client-connectionstress-clkhouse-ppqueu.yaml Success `rm -rf test-db-client-connectionstress-clkhouse-ppqueu.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 47s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 58s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 64s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 70s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 76s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 82s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 1/1 Running 0 88s check pod test-db-client-connectionstress-clkhouse-ppqueu status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-clkhouse-ppqueu 0/1 Completed 0 94s check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done 03:50:49.782 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << HTTP/1.1 200 OK 03:50:49.782 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << Date: Wed, 28 May 2025 03:50:49 GMT 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << Connection: Keep-Alive 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << Content-Type: text/tab-separated-values; charset=UTF-8 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << X-ClickHouse-Server-Display-Name: clkhouse-ppqueu-clickhouse-1.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << Transfer-Encoding: chunked 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << X-ClickHouse-Query-Id: 820888ad-98da-4628-9b5c-ebbb27f97af7 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << X-ClickHouse-Format: TabSeparatedWithNamesAndTypes 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << X-ClickHouse-Timezone: Etc/UTC 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << Keep-Alive: timeout=10, max=9999 03:50:49.783 [main] DEBUG org.apache.http.headers -- http-outgoing-4095 << X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"544","elapsed_ns":"1661174"*** 03:50:49.783 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Connection can be kept alive for 10000 MILLISECONDS 03:50:49.783 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection [id: 4095][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123] can be kept alive for 10.0 seconds 03:50:49.783 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-4095: set socket timeout to 0 03:50:49.783 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection released: [id: 4095][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 1; route allocated: 1 of 500; total allocated: 1 of 10000] 03:50:49.783 [main] DEBUG ru.yandex.clickhouse.ClickHouseDriver -- Creating connection 03:50:49.783 [main] DEBUG ru.yandex.clickhouse.ClickHouseConnectionImpl -- Create a new connection to jdbc:clickhouse://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123/default 03:50:49.783 [main] DEBUG ru.yandex.clickhouse.ClickHouseStatementImpl -- Executing SQL: select timezone(), version() FORMAT TabSeparatedWithNamesAndTypes 03:50:49.783 [main] DEBUG ru.yandex.clickhouse.ClickHouseStatementImpl -- Request url: http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123/?compress=1&query_id=5b06a396-df26-40ed-b30d-0d1ffb032e71&extremes=0&database=default 03:50:49.783 [main] DEBUG org.apache.http.client.protocol.RequestAddCookies -- CookieSpec selected: standard 03:50:49.784 [main] DEBUG org.apache.http.client.protocol.RequestAuthCache -- Re-using cached 'basic' auth scheme for http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123 03:50:49.784 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection request: [route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 0; route allocated: 0 of 500; total allocated: 0 of 10000] 03:50:49.784 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection leased: [id: 4096][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 0; route allocated: 1 of 500; total allocated: 1 of 10000] 03:50:49.784 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Opening connection ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123 03:50:49.784 [main] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator -- Connecting to clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local/10.100.20.46:8123 03:50:49.784 [main] DEBUG org.apache.http.impl.conn.DefaultHttpClientConnectionOperator -- Connection established 172.31.11.66:45546<->10.100.20.46:8123 03:50:49.784 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-4096: set socket timeout to 30000 03:50:49.784 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Executing request POST /?compress=1&query_id=5b06a396-df26-40ed-b30d-0d1ffb032e71&extremes=0&database=default HTTP/1.1 03:50:49.784 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Target auth state: UNCHALLENGED 03:50:49.784 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Proxy auth state: UNCHALLENGED 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> POST /?compress=1&query_id=5b06a396-df26-40ed-b30d-0d1ffb032e71&extremes=0&database=default HTTP/1.1 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> Content-Length: 65 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> Content-Type: text/plain; charset=UTF-8 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> Host: clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> Connection: Keep-Alive 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> User-Agent: Apache-HttpClient/4.5.14 (Java/11.0.16) 03:50:49.784 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 >> Authorization: Basic YWRtaW46ZnRpQnRoajdXMzdsd2daMA== 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "POST /?compress=1&query_id=5b06a396-df26-40ed-b30d-0d1ffb032e71&extremes=0&database=default HTTP/1.1[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "Content-Length: 65[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "Content-Type: text/plain; charset=UTF-8[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "Host: clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "Connection: Keep-Alive[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "User-Agent: Apache-HttpClient/4.5.14 (Java/11.0.16)[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "Authorization: Basic YWRtaW46ZnRpQnRoajdXMzdsd2daMA==[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "[\r][\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "select timezone(), version()[\n]" 03:50:49.784 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 >> "FORMAT TabSeparatedWithNamesAndTypes" 03:50:49.786 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "HTTP/1.1 200 OK[\r][\n]" 03:50:49.786 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "Date: Wed, 28 May 2025 03:50:49 GMT[\r][\n]" 03:50:49.786 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "Connection: Keep-Alive[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "Content-Type: text/tab-separated-values; charset=UTF-8[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "X-ClickHouse-Server-Display-Name: clkhouse-ppqueu-clickhouse-1.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "Transfer-Encoding: chunked[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "X-ClickHouse-Query-Id: 5b06a396-df26-40ed-b30d-0d1ffb032e71[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "X-ClickHouse-Format: TabSeparatedWithNamesAndTypes[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "X-ClickHouse-Timezone: Etc/UTC[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "Keep-Alive: timeout=10, max=9999[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"544","elapsed_ns":"1551074"***[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "6B[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "[0x86]L[0xab][0xc6]![0x12][0x8e][0xe6]***&([0xf6]S[0x82][0x82].[0x0][0x0][0x0]#[0x0][0x0][0x0][0xf0][0x14]timezone()[0x9]version()[\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "String[0x9]String[\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "7:[0xb7]j[0xeb]0a[0xb8][0x1c]0[0xcc][0x95][0xa6][0x9c][0x19][0x2][0x82][0x1d][0x0][0x0][0x0][0x12][0x0][0x0][0x0][0xf0][0x3]Etc/UTC[0x9]24.8.3.59[\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "0[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.wire -- http-outgoing-4096 << "[\r][\n]" 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << HTTP/1.1 200 OK 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << Date: Wed, 28 May 2025 03:50:49 GMT 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << Connection: Keep-Alive 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << Content-Type: text/tab-separated-values; charset=UTF-8 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << X-ClickHouse-Server-Display-Name: clkhouse-ppqueu-clickhouse-1.clkhouse-ppqueu-clickhouse-headless.ns-bcrcc.svc.cluster.local 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << Transfer-Encoding: chunked 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << X-ClickHouse-Query-Id: 5b06a396-df26-40ed-b30d-0d1ffb032e71 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << X-ClickHouse-Format: TabSeparatedWithNamesAndTypes 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << X-ClickHouse-Timezone: Etc/UTC 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << Keep-Alive: timeout=10, max=9999 03:50:49.787 [main] DEBUG org.apache.http.headers -- http-outgoing-4096 << X-ClickHouse-Summary: ***"read_rows":"1","read_bytes":"1","written_rows":"0","written_bytes":"0","total_rows_to_read":"0","result_rows":"1","result_bytes":"544","elapsed_ns":"1551074"*** 03:50:49.787 [main] DEBUG org.apache.http.impl.execchain.MainClientExec -- Connection can be kept alive for 10000 MILLISECONDS 03:50:49.787 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection [id: 4096][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123] can be kept alive for 10.0 seconds 03:50:49.787 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-4096: set socket timeout to 0 03:50:49.787 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection released: [id: 4096][route: ***->http://clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local:8123][total available: 1; route allocated: 1 of 500; total allocated: 1 of 10000] 03:50:49.787 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection manager is shutting down 03:50:49.788 [main] DEBUG org.apache.http.impl.conn.DefaultManagedHttpClientConnection -- http-outgoing-0: Close connection 03:50:49.788 [main] DEBUG org.apache.http.impl.conn.PoolingHttpClientConnectionManager -- Connection manager shut down Test Result: null Connection Information: Database Type: clickhouse Host: clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local Port: 8123 Database: default Table: User: admin Org: Access Mode: mysql Test Type: connectionstress Connection Count: 4096 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-clkhouse-ppqueu --namespace ns-bcrcc ` pod/test-db-client-connectionstress-clkhouse-ppqueu patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-clkhouse-ppqueu" force deleted check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ppqueu" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-clkhouse-ppqueu.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-clkhouse-ppqueu created apply test-chaos-mesh-dnsrandom-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-dnsrandom-clkhouse-ppqueu.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:39 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ppqueu" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover drainnode check node drain check node drain success kubectl get pod clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-4-183.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod clkhouse-ppqueu-clickhouse-1 --namespace ns-bcrcc -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-3-65.us-west-2.compute.internal success kubectl drain ip-172-31-4-183.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-4-183.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-pjwt6, kube-system/aws-node-ltjjc, kube-system/ebs-csi-node-bb94s, kube-system/kube-proxy-fr7dg evicting pod ns-wptnn/pulsarm-qtxszq-zookeeper-0 evicting pod ns-bcrcc/clkhouse-ppqueu-clickhouse-0 evicting pod kube-system/ebs-csi-controller-9df898bbf-smvzw evicting pod ns-wptnn/pulsarm-qtxszq-bookies-2 pod/pulsarm-qtxszq-bookies-2 evicted pod/pulsarm-qtxszq-zookeeper-0 evicted pod/clkhouse-ppqueu-clickhouse-0 evicted pod/ebs-csi-controller-9df898bbf-smvzw evicted node/ip-172-31-4-183.us-west-2.compute.internal drained kubectl uncordon ip-172-31-4-183.us-west-2.compute.internal node/ip-172-31-4-183.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:54 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 11:40 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover drainnode Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart clkhouse-ppqueu --auto-approve --force=true --components clickhouse --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-restart-9h9vq created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-restart-9h9vq -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-restart-9h9vq ns-bcrcc Restart clkhouse-ppqueu clickhouse Running 0/2 May 28,2025 11:55 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-restart-9h9vq ns-bcrcc Restart clkhouse-ppqueu clickhouse Succeed 2/2 May 28,2025 11:55 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-restart-9h9vq ns-bcrcc Restart clkhouse-ppqueu clickhouse Succeed 2/2 May 28,2025 11:55 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-restart-9h9vq --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-restart-9h9vq patched `kbcli cluster delete-ops --name clkhouse-ppqueu-restart-9h9vq --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-restart-9h9vq deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ppqueu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-clkhouse-ppqueu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-clkhouse-ppqueu created apply test-chaos-mesh-networkduplicate-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-networkduplicate-clkhouse-ppqueu.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:20Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ppqueu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success check component ch-keeper exists `kubectl get components -l app.kubernetes.io/instance=clkhouse-ppqueu,apps.kubeblocks.io/component-name=ch-keeper --namespace ns-bcrcc | (grep "ch-keeper" || true )` `kubectl get pvc -l app.kubernetes.io/instance=clkhouse-ppqueu,apps.kubeblocks.io/component-name=ch-keeper,apps.kubeblocks.io/vct-name=data --namespace ns-bcrcc ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ppqueu namespace. `kbcli cluster volume-expand clkhouse-ppqueu --auto-approve --force=true --components ch-keeper --volume-claim-templates data --storage 22Gi --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-volumeexpansion-r95m7 created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-volumeexpansion-r95m7 -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-volumeexpansion-r95m7 ns-bcrcc VolumeExpansion clkhouse-ppqueu ch-keeper Running 0/1 May 28,2025 12:00 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done No resources found in clkhouse-ppqueu namespace. check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-volumeexpansion-r95m7 ns-bcrcc VolumeExpansion clkhouse-ppqueu ch-keeper Succeed 1/1 May 28,2025 12:00 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-volumeexpansion-r95m7 ns-bcrcc VolumeExpansion clkhouse-ppqueu ch-keeper Succeed 1/1 May 28,2025 12:00 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-volumeexpansion-r95m7 --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-volumeexpansion-r95m7 patched `kbcli cluster delete-ops --name clkhouse-ppqueu-volumeexpansion-r95m7 --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-volumeexpansion-r95m7 deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ppqueu" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-clkhouse-ppqueu.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-clkhouse-ppqueu created apply test-chaos-mesh-podfailure-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-podfailure-clkhouse-ppqueu.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ppqueu" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover podfailure Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ppqueu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-clkhouse-ppqueu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-clkhouse-ppqueu created apply test-chaos-mesh-networklossover-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-networklossover-clkhouse-ppqueu.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ppqueu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ppqueu" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-clkhouse-ppqueu.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-clkhouse-ppqueu created apply test-chaos-mesh-timeoffset-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-timeoffset-clkhouse-ppqueu.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ppqueu" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ppqueu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 action: partition mode: all target: mode: all selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-clkhouse-ppqueu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-clkhouse-ppqueu created apply test-chaos-mesh-networkpartition-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-networkpartition-clkhouse-ppqueu.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ppqueu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu created apply test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover networkbandwidthover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ppqueu" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-clkhouse-ppqueu.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-clkhouse-ppqueu created apply test-chaos-mesh-networkcorruptover-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-clkhouse-ppqueu.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 11:56 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-clkhouse-ppqueu" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-clkhouse-ppqueu patched check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover networkcorruptover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover delete pod:clkhouse-ppqueu-clickhouse-0 `kubectl delete pod clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc ` pod "clkhouse-ppqueu-clickhouse-0" deleted check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 11:47 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:17 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 11:55 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "CREATE TABLE test_kbcli (id Int32,name String) ENGINE = MergeTree() ORDER BY id;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` `echo "clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password \"ftiBthj7W37lwgZ0\" --query \"INSERT INTO test_kbcli VALUES (1,'nevso');\" " | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Received exception from server (version 24.8.3): Code: 60. DB::Exception: Received from clkhouse-ppqueu-clickhouse:9000. DB::Exception: Table default.test_kbcli does not exist. (UNKNOWN_TABLE) (query: INSERT INTO test_kbcli VALUES (1,'nevso');) command terminated with exit code 60 `echo "clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password \"ftiBthj7W37lwgZ0\" --query \"INSERT INTO test_kbcli VALUES (1,'nevso');\" " | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file `clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT * FROM test_kbcli;"` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg: Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg: Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return msg:1 nevso check msg:[nevso] equal msg:[1 nevso] cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart clkhouse-ppqueu --auto-approve --force=true --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-restart-ldcf5 created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-restart-ldcf5 -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-restart-ldcf5 ns-bcrcc Restart clkhouse-ppqueu clickhouse,ch-keeper Running 0/3 May 28,2025 12:19 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:20 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:21 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:20 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-restart-ldcf5 ns-bcrcc Restart clkhouse-ppqueu clickhouse,ch-keeper Succeed 3/3 May 28,2025 12:19 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-restart-ldcf5 ns-bcrcc Restart clkhouse-ppqueu clickhouse,ch-keeper Succeed 3/3 May 28,2025 12:19 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-restart-ldcf5 --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-restart-ldcf5 patched `kbcli cluster delete-ops --name clkhouse-ppqueu-restart-ldcf5 --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-restart-ldcf5 deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop clkhouse-ppqueu --auto-approve --force=true --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-stop-x9qpd created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-stop-x9qpd -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-stop-x9qpd ns-bcrcc Stop clkhouse-ppqueu ch-keeper,clickhouse Running 0/3 May 28,2025 12:22 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Stopped May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-stop-x9qpd ns-bcrcc Stop clkhouse-ppqueu ch-keeper,clickhouse Succeed 3/3 May 28,2025 12:22 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-stop-x9qpd ns-bcrcc Stop clkhouse-ppqueu ch-keeper,clickhouse Succeed 3/3 May 28,2025 12:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-stop-x9qpd --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-stop-x9qpd patched `kbcli cluster delete-ops --name clkhouse-ppqueu-stop-x9qpd --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-stop-x9qpd deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start clkhouse-ppqueu --force=true --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-start-4lfvs created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-start-4lfvs -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-start-4lfvs ns-bcrcc Start clkhouse-ppqueu ch-keeper,clickhouse Running 0/3 May 28,2025 12:22 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:23 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-start-4lfvs ns-bcrcc Start clkhouse-ppqueu ch-keeper,clickhouse Succeed 3/3 May 28,2025 12:22 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-start-4lfvs ns-bcrcc Start clkhouse-ppqueu ch-keeper,clickhouse Succeed 3/3 May 28,2025 12:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-start-4lfvs --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-start-4lfvs patched `kbcli cluster delete-ops --name clkhouse-ppqueu-start-4lfvs --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-start-4lfvs deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ppqueu" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-clkhouse-ppqueu.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-clkhouse-ppqueu created apply test-chaos-mesh-dnserror-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-dnserror-clkhouse-ppqueu.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:23 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ppqueu" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-clkhouse-ppqueu" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-clkhouse-ppqueu.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-clkhouse-ppqueu created apply test-chaos-mesh-oom-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-oom-clkhouse-ppqueu.yaml` check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:23 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-clkhouse-ppqueu" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-clkhouse-ppqueu patched check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover oom Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success cluster clickhouse scale-out check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ppqueu namespace. `kbcli cluster scale-out clkhouse-ppqueu --auto-approve --force=true --components clickhouse --replicas 1 --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-horizontalscaling-s29gl created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-horizontalscaling-s29gl -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-horizontalscaling-s29gl ns-bcrcc HorizontalScaling clkhouse-ppqueu clickhouse Running 0/1 May 28,2025 12:27 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:23 UTC+0800 clkhouse-ppqueu-clickhouse-2 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-13-166.us-west-2.compute.internal/172.31.13.166 May 28,2025 12:27 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done No resources found in clkhouse-ppqueu namespace. check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-horizontalscaling-s29gl ns-bcrcc HorizontalScaling clkhouse-ppqueu clickhouse Succeed 1/1 May 28,2025 12:27 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-horizontalscaling-s29gl ns-bcrcc HorizontalScaling clkhouse-ppqueu clickhouse Succeed 1/1 May 28,2025 12:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-horizontalscaling-s29gl --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-horizontalscaling-s29gl patched `kbcli cluster delete-ops --name clkhouse-ppqueu-horizontalscaling-s29gl --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-horizontalscaling-s29gl deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success cluster clickhouse scale-in check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ppqueu namespace. `kbcli cluster scale-in clkhouse-ppqueu --auto-approve --force=true --components clickhouse --replicas 1 --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-horizontalscaling-rxnld created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-horizontalscaling-rxnld -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-horizontalscaling-rxnld ns-bcrcc HorizontalScaling clkhouse-ppqueu clickhouse Running 0/1 May 28,2025 12:28 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:20Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:23 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done No resources found in clkhouse-ppqueu namespace. check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-horizontalscaling-rxnld ns-bcrcc HorizontalScaling clkhouse-ppqueu clickhouse Succeed 1/1 May 28,2025 12:28 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-horizontalscaling-rxnld ns-bcrcc HorizontalScaling clkhouse-ppqueu clickhouse Succeed 1/1 May 28,2025 12:28 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-horizontalscaling-rxnld --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-horizontalscaling-rxnld patched `kbcli cluster delete-ops --name clkhouse-ppqueu-horizontalscaling-rxnld --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-horizontalscaling-rxnld deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=clkhouse-ppqueu,apps.kubeblocks.io/component-name=clickhouse,apps.kubeblocks.io/vct-name=data --namespace ns-bcrcc ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in clkhouse-ppqueu namespace. `kbcli cluster volume-expand clkhouse-ppqueu --auto-approve --force=true --components clickhouse --volume-claim-templates data --storage 24Gi --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-volumeexpansion-tb2kg created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-volumeexpansion-tb2kg -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-volumeexpansion-tb2kg ns-bcrcc VolumeExpansion clkhouse-ppqueu clickhouse Running 0/2 May 28,2025 12:29 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:24Gi ip-172-31-3-65.us-west-2.compute.internal/172.31.3.65 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 200m / 200m 1Gi / 1Gi data:24Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:23 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done No resources found in clkhouse-ppqueu namespace. check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-volumeexpansion-tb2kg ns-bcrcc VolumeExpansion clkhouse-ppqueu clickhouse Succeed 2/2 May 28,2025 12:29 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-volumeexpansion-tb2kg ns-bcrcc VolumeExpansion clkhouse-ppqueu clickhouse Succeed 2/2 May 28,2025 12:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-volumeexpansion-tb2kg --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-volumeexpansion-tb2kg patched `kbcli cluster delete-ops --name clkhouse-ppqueu-volumeexpansion-tb2kg --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-volumeexpansion-tb2kg deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale clkhouse-ppqueu --auto-approve --force=true --components clickhouse --cpu 300m --memory 1.1Gi --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-verticalscaling-nsk79 created successfully, you can view the progress: kbcli cluster describe-ops clkhouse-ppqueu-verticalscaling-nsk79 -n ns-bcrcc check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-verticalscaling-nsk79 ns-bcrcc VerticalScaling clkhouse-ppqueu clickhouse Running 0/2 May 28,2025 12:31 UTC+0800 check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Updating May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:24Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 12:32 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:24Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 12:31 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done check ops status `kbcli cluster list-ops clkhouse-ppqueu --status all --namespace ns-bcrcc ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME clkhouse-ppqueu-verticalscaling-nsk79 ns-bcrcc VerticalScaling clkhouse-ppqueu clickhouse Succeed 2/2 May 28,2025 12:31 UTC+0800 check ops status done ops_status:clkhouse-ppqueu-verticalscaling-nsk79 ns-bcrcc VerticalScaling clkhouse-ppqueu clickhouse Succeed 2/2 May 28,2025 12:31 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations clkhouse-ppqueu-verticalscaling-nsk79 --namespace ns-bcrcc ` opsrequest.operations.kubeblocks.io/clkhouse-ppqueu-verticalscaling-nsk79 patched `kbcli cluster delete-ops --name clkhouse-ppqueu-verticalscaling-nsk79 --force --auto-approve --namespace ns-bcrcc ` OpsRequest clkhouse-ppqueu-verticalscaling-nsk79 deleted `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ppqueu" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ppqueu" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-clkhouse-ppqueu namespace: ns-bcrcc spec: selector: namespaces: - ns-bcrcc labelSelectors: apps.kubeblocks.io/pod-name: clkhouse-ppqueu-clickhouse-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-clkhouse-ppqueu.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-clkhouse-ppqueu created apply test-chaos-mesh-fullcpu-clkhouse-ppqueu.yaml Success `rm -rf test-chaos-mesh-fullcpu-clkhouse-ppqueu.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:24Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 12:32 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:24Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 12:31 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-clkhouse-ppqueu --namespace ns-bcrcc ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ppqueu" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-clkhouse-ppqueu" not found check failover pod name failover pod name:clkhouse-ppqueu-clickhouse-0 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check db_client batch data count `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse --port 9000 --user admin --password "ftiBthj7W37lwgZ0" --query "SELECT count(*) FROM executions_loop.executions_loop_table;"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update clkhouse-ppqueu --termination-policy=WipeOut --namespace ns-bcrcc ` cluster.apps.kubeblocks.io/clkhouse-ppqueu updated (no change) check cluster status `kbcli cluster list clkhouse-ppqueu --show-labels --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS clkhouse-ppqueu ns-bcrcc clickhouse WipeOut Running May 28,2025 11:39 UTC+0800 app.kubernetes.io/instance=clkhouse-ppqueu,clusterdefinition.kubeblocks.io/name=clickhouse check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances clkhouse-ppqueu --namespace ns-bcrcc ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME clkhouse-ppqueu-ch-keeper-0 ns-bcrcc clkhouse-ppqueu ch-keeper Running leader us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:22Gi ip-172-31-9-236.us-west-2.compute.internal/172.31.9.236 May 28,2025 12:22 UTC+0800 clkhouse-ppqueu-clickhouse-0 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:24Gi ip-172-31-2-57.us-west-2.compute.internal/172.31.2.57 May 28,2025 12:32 UTC+0800 clkhouse-ppqueu-clickhouse-1 ns-bcrcc clkhouse-ppqueu clickhouse Running us-west-2a 300m / 300m 1181116006400m / 1181116006400m data:24Gi ip-172-31-4-183.us-west-2.compute.internal/172.31.4.183 May 28,2025 12:31 UTC+0800 check pod status done `kubectl get secrets -l app.kubernetes.io/instance=clkhouse-ppqueu` set secret: clkhouse-ppqueu-clickhouse-account-admin `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.username***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.password***"` `kubectl get secrets clkhouse-ppqueu-clickhouse-account-admin -o jsonpath="***.data.port***"` DB_USERNAME_PROXY:;DB_PASSWORD_PROXY:;DB_PORT_PROXY:8123 DB_USERNAME:admin;DB_PASSWORD:ftiBthj7W37lwgZ0;DB_PORT:9000;DB_DATABASE:default check cluster connect `echo 'clickhouse-client --host clkhouse-ppqueu-clickhouse.ns-bcrcc.svc.cluster.local --port 9000 --user admin --password "ftiBthj7W37lwgZ0"' | kubectl exec -it clkhouse-ppqueu-clickhouse-0 --namespace ns-bcrcc -- bash` check cluster connect done cluster list-logs `kbcli cluster list-logs clkhouse-ppqueu --namespace ns-bcrcc ` No log files found. Error from server (NotFound): pods "clkhouse-ppqueu-clickhouse-0" not found cluster logs `kbcli cluster logs clkhouse-ppqueu --tail 30 --namespace ns-bcrcc ` Defaulted container "clickhouse" out of: clickhouse, kbagent, init-kbagent (init), kbagent-worker (init) 2025.05.28 04:34:49.377947 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9996 ms 2025.05.28 04:34:50.751389 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9998 ms 2025.05.28 04:34:55.529178 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9985 ms 2025.05.28 04:34:59.378143 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9997 ms 2025.05.28 04:35:00.751380 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9996 ms 2025.05.28 04:35:05.529203 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9997 ms 2025.05.28 04:35:09.378363 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9996 ms 2025.05.28 04:35:10.751389 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9997 ms 2025.05.28 04:35:15.529223 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9998 ms 2025.05.28 04:35:19.380824 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9999 ms 2025.05.28 04:35:20.751394 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9996 ms 2025.05.28 04:35:25.529245 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9997 ms 2025.05.28 04:35:29.378208 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9994 ms 2025.05.28 04:35:30.751384 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9997 ms 2025.05.28 04:35:35.529226 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9997 ms 2025.05.28 04:35:39.378205 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9997 ms 2025.05.28 04:35:40.751432 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9997 ms 2025.05.28 04:35:45.529298 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9997 ms 2025.05.28 04:35:49.378187 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9997 ms 2025.05.28 04:35:50.751373 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9997 ms 2025.05.28 04:35:55.529298 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9996 ms 2025.05.28 04:35:59.378252 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9996 ms 2025.05.28 04:36:00.751403 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9996 ms 2025.05.28 04:36:05.529232 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9997 ms 2025.05.28 04:36:09.378296 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 9997 ms 2025.05.28 04:36:10.751372 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9997 ms 2025.05.28 04:36:15.540244 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 10009 ms 2025.05.28 04:36:19.440266 [ 54 ] *** KeeperTCPHandler: Polling socket for session 25 took 10058 ms 2025.05.28 04:36:20.751387 [ 56 ] *** KeeperTCPHandler: Polling socket for session 21 took 9998 ms 2025.05.28 04:36:25.529253 [ 57 ] *** KeeperTCPHandler: Polling socket for session 26 took 9985 ms delete cluster clkhouse-ppqueu `kbcli cluster delete clkhouse-ppqueu --auto-approve --namespace ns-bcrcc ` Cluster clkhouse-ppqueu deleted pod_info:clkhouse-ppqueu-ch-keeper-0 2/2 Terminating 0 13m clkhouse-ppqueu-clickhouse-0 2/2 Terminating 0 4m4s clkhouse-ppqueu-clickhouse-1 2/2 Terminating 0 4m43s No resources found in ns-bcrcc namespace. delete cluster pod done No resources found in ns-bcrcc namespace. check cluster resource non-exist OK: pvc No resources found in ns-bcrcc namespace. delete cluster done No resources found in ns-bcrcc namespace. No resources found in ns-bcrcc namespace. No resources found in ns-bcrcc namespace. Clickhouse Test Suite All Done! --------------------------------------Clickhouse (Topology = cluster Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=clickhouse-24-1.0.0-alpha.0;ComponentVersion=clickhouse;ServiceVersion=24.8.3;]|[Description=Create a cluster with the specified component definition clickhouse-24-1.0.0-alpha.0 and component version clickhouse and service version 24.8.3] [PASSED]|[Connect]|[ComponentName=clickhouse]|[Description=Connect to the cluster] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[VerticalScaling]|[ComponentName=ch-keeper]|[Description=VerticalScaling the cluster specify component ch-keeper] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=clickhouse]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=clickhouse]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Restart]|[ComponentName=clickhouse]|[Description=Restart the cluster specify component clickhouse] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[VolumeExpansion]|[ComponentName=ch-keeper]|[Description=VolumeExpansion the cluster specify component ch-keeper] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=clickhouse]|[Description=] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=clickhouse]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=clickhouse]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=clickhouse]|[Description=] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=clickhouse]|[Description=] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=clickhouse]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[HorizontalScaling Out]|[ComponentName=clickhouse]|[Description=HorizontalScaling Out the cluster specify component clickhouse] [PASSED]|[HorizontalScaling In]|[ComponentName=clickhouse]|[Description=HorizontalScaling In the cluster specify component clickhouse] [PASSED]|[VolumeExpansion]|[ComponentName=clickhouse]|[Description=VolumeExpansion the cluster specify component clickhouse] [PASSED]|[VerticalScaling]|[ComponentName=clickhouse]|[Description=VerticalScaling the cluster specify component clickhouse] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=clickhouse]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]