source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-wscoz ` `kubectl create namespace ns-wscoz` namespace/ns-wscoz created create namespace ns-wscoz done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "1.0" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v1.0.0` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 33.6M 100 33.6M 0 0 121M 0 --:--:-- --:--:-- --:--:-- 121M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v1.0.0 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 1.0.0 kbcli: 1.0.0 Kubernetes Env: v1.32.5-eks-5d4a308 check snapshot controller check snapshot controller done eks default-vsc found POD_RESOURCES: No resources found found default storage class: gp3 KubeBlocks version is:1.0.0 skip upgrade KubeBlocks current KubeBlocks version: 1.0.0 Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check component definition set component name:kafka-combine set component version set component version:kafka-broker set service versions:3.3.2,2.7.0 set service versions sorted:2.7.0,3.3.2 set replicas first:3,3.3.2 set replicas third:3,3.3.2 set replicas fourth:3,3.3.2 set minimum cmpv service version set minimum cmpv service version replicas:3,3.3.2 REPORT_COUNT:1 CLUSTER_TOPOLOGY:separated_monitor topology separated_monitor found in cluster definition kafka LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 1 No resources found in ns-wscoz namespace. termination_policy:DoNotTerminate create 3 replica DoNotTerminate kafka cluster check component definition set component definition by component version check cmpd by labels check cmpd by compDefs set component definition: kafka-broker-1.0.0-alpha.0 by component version:kafka-broker apiVersion: apps.kubeblocks.io/v1 kind: Cluster metadata: name: kafka-htlrnp namespace: ns-wscoz annotations: "kubeblocks.io/extra-env": '***"KB_KAFKA_ENABLE_SASL":"false","KB_KAFKA_BROKER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_CONTROLLER_HEAP":"-XshowSettings:vm -XX:MaxRAMPercentage=100 -Ddepth=64","KB_KAFKA_PUBLIC_ACCESS":"false"***' spec: clusterDef: kafka topology: separated_monitor terminationPolicy: DoNotTerminate componentSpecs: - name: kafka-broker tls: false disableExporter: true replicas: 3 serviceVersion: 3.3.2 services: - name: advertised-listener serviceType: ClusterIP podService: true resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi env: - name: KB_BROKER_DIRECT_POD_ACCESS value: "false" - name: KB_KAFKA_ENABLE_SASL_SCRAM value: "false" volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: metadata spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: kafka-controller tls: false replicas: 3 serviceVersion: 3.3.2 disableExporter: true resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: metadata spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi - name: kafka-exporter replicas: 1 disableExporter: true env: - name: KB_KAFKA_ENABLE_SASL_SCRAM value: "false" resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi `kubectl apply -f test_create_kafka-htlrnp.yaml` cluster.apps.kubeblocks.io/kafka-htlrnp created apply test_create_kafka-htlrnp.yaml Success `rm -rf test_create_kafka-htlrnp.yaml` check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Creating May 28,2025 11:36 UTC+0800 clusterdefinition.kubeblocks.io/name=kafka cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:38 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl get secrets -l app.kubernetes.io/instance=kafka-htlrnp` No resources found in ns-wscoz namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:9092;DB_DATABASE: There is no password in Type: 7. get cluster component name set component name:kafka-broker describe cluster `kbcli cluster describe kafka-htlrnp --namespace ns-wscoz ` Name: kafka-htlrnp Created Time: May 28,2025 11:36 UTC+0800 NAMESPACE CLUSTER-DEFINITION TOPOLOGY STATUS TERMINATION-POLICY ns-wscoz kafka separated_monitor Running DoNotTerminate Endpoints: COMPONENT INTERNAL EXTERNAL kafka-broker kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local:9092 kafka-htlrnp-kafka-broker-advertised-listener-1.ns-wscoz.svc.cluster.local:9092 kafka-htlrnp-kafka-broker-advertised-listener-2.ns-wscoz.svc.cluster.local:9092 Topology: COMPONENT SERVICE-VERSION INSTANCE ROLE STATUS AZ NODE CREATED-TIME kafka-broker 3.3.2 kafka-htlrnp-kafka-broker-0 Running us-west-2a ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:38 UTC+0800 kafka-broker 3.3.2 kafka-htlrnp-kafka-broker-1 Running us-west-2a ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 kafka-broker 3.3.2 kafka-htlrnp-kafka-broker-2 Running us-west-2a ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 kafka-controller 3.3.2 kafka-htlrnp-kafka-controller-0 Running us-west-2a ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-controller 3.3.2 kafka-htlrnp-kafka-controller-1 Running us-west-2a ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-controller 3.3.2 kafka-htlrnp-kafka-controller-2 Running us-west-2a ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-exporter 1.6.0 kafka-htlrnp-kafka-exporter-0 Running us-west-2a ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 Resources Allocation: COMPONENT INSTANCE-TEMPLATE CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS kafka-controller 500m / 500m 512Mi / 512Mi metadata:1Gi kb-default-sc kafka-broker 500m / 500m 512Mi / 512Mi data:1Gi kb-default-sc metadata:1Gi kb-default-sc kafka-exporter 500m / 500m 512Mi / 512Mi Images: COMPONENT COMPONENT-DEFINITION IMAGE kafka-controller kafka-controller-1.0.0-alpha.0 docker.io/apecloud/kafka:3.3.2-debian-11-r54 docker.io/apecloud/jmx-exporter:0.18.0-debian-11-r20 kafka-broker kafka-broker-1.0.0-alpha.0 docker.io/apecloud/kafka:3.3.2-debian-11-r54 docker.io/apecloud/jmx-exporter:0.18.0-debian-11-r20 kafka-exporter kafka-exporter-1.0.0-alpha.0 docker.io/apecloud/kafka-exporter:1.6.0-debian-11-r67 Show cluster events: kbcli cluster list-events -n ns-wscoz kafka-htlrnp `kbcli cluster label kafka-htlrnp app.kubernetes.io/instance- --namespace ns-wscoz ` label "app.kubernetes.io/instance" not found. `kbcli cluster label kafka-htlrnp app.kubernetes.io/instance=kafka-htlrnp --namespace ns-wscoz ` `kbcli cluster label kafka-htlrnp --list --namespace ns-wscoz ` NAME NAMESPACE LABELS kafka-htlrnp ns-wscoz app.kubernetes.io/instance=kafka-htlrnp clusterdefinition.kubeblocks.io/name=kafka label cluster app.kubernetes.io/instance=kafka-htlrnp Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=kafka-htlrnp --namespace ns-wscoz ` `kbcli cluster label kafka-htlrnp --list --namespace ns-wscoz ` NAME NAMESPACE LABELS kafka-htlrnp ns-wscoz app.kubernetes.io/instance=kafka-htlrnp case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=kafka label cluster case.name=kbcli.test1 Success `kbcli cluster label kafka-htlrnp case.name=kbcli.test2 --overwrite --namespace ns-wscoz ` `kbcli cluster label kafka-htlrnp --list --namespace ns-wscoz ` NAME NAMESPACE LABELS kafka-htlrnp ns-wscoz app.kubernetes.io/instance=kafka-htlrnp case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=kafka label cluster case.name=kbcli.test2 Success `kbcli cluster label kafka-htlrnp case.name- --namespace ns-wscoz ` `kbcli cluster label kafka-htlrnp --list --namespace ns-wscoz ` NAME NAMESPACE LABELS kafka-htlrnp ns-wscoz app.kubernetes.io/instance=kafka-htlrnp clusterdefinition.kubeblocks.io/name=kafka delete cluster label case.name Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-kafka-htlrnp" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-kafka-htlrnp --namespace ns-wscoz ` Error from server (NotFound): pods "test-db-client-executionloop-kafka-htlrnp" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-kafka-htlrnp" not found `kubectl get secrets -l app.kubernetes.io/instance=kafka-htlrnp` No resources found in ns-wscoz namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:9092;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-kafka-htlrnp namespace: ns-wscoz spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local" - "--user" - "" - "--password" - "" - "--port" - "9092" - "--dbtype" - "kafka" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-kafka-htlrnp.yaml` pod/test-db-client-executionloop-kafka-htlrnp created apply test-db-client-executionloop-kafka-htlrnp.yaml Success `rm -rf test-db-client-executionloop-kafka-htlrnp.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 39s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 1/1 Running 0 63s check pod test-db-client-executionloop-kafka-htlrnp status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-kafka-htlrnp 0/1 Completed 0 68s check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:38 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 03:43:36.073 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34934): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34925, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.073 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34935) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.074 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34935): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34926, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.074 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34936) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.075 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34936): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34927, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.075 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34937) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.076 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34937): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34928, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.076 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34938) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.076 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34938): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34929, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.077 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34939) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.077 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34939): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34930, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.077 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34940) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.078 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34940): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34931, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.078 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34941) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.079 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34941): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34932, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.079 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34942) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.080 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34942): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34933, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.080 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34943) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.081 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34943): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34934, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.081 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34944) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.082 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34944): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34935, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.082 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34945) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.083 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34945): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34936, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.083 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34946) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.084 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34946): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34937, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.084 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34947) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.084 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34947): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34938, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.085 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34948) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.085 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34948): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34939, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.086 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34949) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.086 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34949): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34940, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.086 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34950) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.087 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34950): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34941, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.087 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34951) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.087 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34951): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34942, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.088 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34952) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.088 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34952): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34943, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.088 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34953) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.089 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34953): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34944, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.089 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34954) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.089 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34954): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34945, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.090 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34955) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.090 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34955): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34946, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.090 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34956) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.091 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34956): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34947, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.091 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34957) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.091 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34957): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34948, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.092 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34958) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.092 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34958): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34949, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.092 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34959) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.093 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34959): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34950, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.093 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34960) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.093 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34960): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34951, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) 03:43:36.094 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Sending PRODUCE request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34961) and timeout 30000 to node 102: ***acks=1,timeout=30000,partitionSizes=[executions_loop_topic-0=120]*** 03:43:36.094 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.NetworkClient -- [Producer clientId=producer-1] Received PRODUCE response from node 102 for request with header RequestHeader(apiKey=PRODUCE, apiVersion=9, clientId=producer-1, correlationId=34961): ProduceResponseData(responses=[TopicProduceResponse(name='executions_loop_topic', partitionResponses=[PartitionProduceResponse(index=0, errorCode=0, baseOffset=34952, logAppendTimeMs=-1, logStartOffset=0, recordErrors=[], errorMessage=null)])], throttleTimeMs=0) [ 60s ] executions total: 34953 successful: 34953 failed: 0 disconnect: 0 03:43:36.095 [main] INFO org.apache.kafka.clients.producer.KafkaProducer -- [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. 03:43:36.096 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.producer.internals.Sender -- [Producer clientId=producer-1] Beginning shutdown of Kafka producer I/O thread, sending remaining records. 03:43:36.102 [kafka-producer-network-thread | producer-1] DEBUG org.apache.kafka.clients.producer.internals.Sender -- [Producer clientId=producer-1] Shutdown of Kafka producer I/O thread has completed. 03:43:36.103 [main] INFO org.apache.kafka.common.metrics.Metrics -- Metrics scheduler closed 03:43:36.103 [main] INFO org.apache.kafka.common.metrics.Metrics -- Closing reporter org.apache.kafka.common.metrics.JmxReporter 03:43:36.103 [main] INFO org.apache.kafka.common.metrics.Metrics -- Metrics reporters closed 03:43:36.103 [main] INFO org.apache.kafka.common.utils.AppInfoParser -- App info kafka.producer for producer-1 unregistered 03:43:36.103 [main] DEBUG org.apache.kafka.clients.producer.KafkaProducer -- [Producer clientId=producer-1] Kafka producer has been closed 03:43:36.105 [main] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator -- [Consumer clientId=consumer-kafka-tester-group-1, groupId=kafka-tester-group] Sending synchronous auto-commit of offsets *** 03:43:36.105 [main] DEBUG org.apache.kafka.clients.consumer.internals.ConsumerCoordinator -- [Consumer clientId=consumer-kafka-tester-group-1, groupId=kafka-tester-group] Executing onLeavePrepare with generation Generation***generationId=-1, memberId='', protocol='null'*** and memberId 03:43:36.105 [main] DEBUG org.apache.kafka.clients.consumer.internals.AbstractCoordinator -- [Consumer clientId=consumer-kafka-tester-group-1, groupId=kafka-tester-group] Resetting generation due to consumer pro-actively leaving the group 03:43:36.106 [main] INFO org.apache.kafka.common.metrics.Metrics -- Metrics scheduler closed 03:43:36.106 [main] INFO org.apache.kafka.common.metrics.Metrics -- Closing reporter org.apache.kafka.common.metrics.JmxReporter 03:43:36.106 [main] INFO org.apache.kafka.common.metrics.Metrics -- Metrics reporters closed 03:43:36.108 [main] INFO org.apache.kafka.common.utils.AppInfoParser -- App info kafka.consumer for consumer-kafka-tester-group-1 unregistered 03:43:36.108 [main] DEBUG org.apache.kafka.clients.consumer.KafkaConsumer -- [Consumer clientId=consumer-kafka-tester-group-1, groupId=kafka-tester-group] Kafka consumer has been closed 03:43:36.108 [main] DEBUG org.apache.kafka.clients.admin.KafkaAdminClient -- [AdminClient clientId=adminclient-1] Initiating close operation. 03:43:36.108 [main] DEBUG org.apache.kafka.clients.admin.KafkaAdminClient -- [AdminClient clientId=adminclient-1] Waiting for the I/O thread to exit. Hard shutdown in 31536000000 ms. 03:43:36.109 [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.utils.AppInfoParser -- App info kafka.admin.client for adminclient-1 unregistered 03:43:36.111 [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.metrics.Metrics -- Metrics scheduler closed 03:43:36.111 [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.metrics.Metrics -- Closing reporter org.apache.kafka.common.metrics.JmxReporter 03:43:36.112 [kafka-admin-client-thread | adminclient-1] INFO org.apache.kafka.common.metrics.Metrics -- Metrics reporters closed 03:43:36.112 [kafka-admin-client-thread | adminclient-1] DEBUG org.apache.kafka.clients.admin.KafkaAdminClient -- [AdminClient clientId=adminclient-1] Exiting AdminClientRunnable thread. 03:43:36.112 [main] DEBUG org.apache.kafka.clients.admin.KafkaAdminClient -- [AdminClient clientId=adminclient-1] Kafka admin client closed. Test Result: Total Executions: 34953 Successful Executions: 34953 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: kafka Host: kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local Port: 9092 Database: Table: User: Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds Topic: DB_CLIENT_BATCH_DATA_COUNT: 34953 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-kafka-htlrnp --namespace ns-wscoz ` pod/test-db-client-executionloop-kafka-htlrnp patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-kafka-htlrnp" force deleted test failover fullcpu check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kafka-htlrnp" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-kafka-htlrnp.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-kafka-htlrnp created apply test-chaos-mesh-fullcpu-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-fullcpu-kafka-htlrnp.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:38 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kafka-htlrnp" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover fullcpu Success No resources found in ns-wscoz namespace. `kubectl get secrets -l app.kubernetes.io/instance=kafka-htlrnp` Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:9092;DB_DATABASE: No resources found in ns-wscoz namespace. No resources found in ns-wscoz namespace. create kafka-producer pod kafka-htlrnp-kafka-producer `kubectl run --namespace ns-wscoz kafka-htlrnp-kafka-producer --restart='Never' --image docker.io/apecloud/kafka:3.3.2-debian-11-r54 --command -- sleep infinity` pod/kafka-htlrnp-kafka-producer created check pod status pod_status:NAME READY STATUS RESTARTS AGE kafka-htlrnp-kafka-producer 0/1 ContainerCreating 0 1s check pod kafka-htlrnp-kafka-producer status done `kubectl wait --for=condition=Ready pod kafka-htlrnp-kafka-producer --namespace ns-wscoz` pod/kafka-htlrnp-kafka-producer condition met pod_status:NAME READY STATUS RESTARTS AGE kafka-htlrnp-kafka-producer 1/1 Running 0 6s create kafka-consumer pod kafka-htlrnp-kafka-consumer `kubectl run --namespace ns-wscoz kafka-htlrnp-kafka-consumer --restart='Never' --image docker.io/apecloud/kafka:3.3.2-debian-11-r54 --command -- sleep infinity` pod/kafka-htlrnp-kafka-consumer created check pod status pod_status:NAME READY STATUS RESTARTS AGE kafka-htlrnp-kafka-consumer 0/1 ContainerCreating 0 0s check pod kafka-htlrnp-kafka-consumer status done `kubectl wait --for=condition=Ready pod kafka-htlrnp-kafka-consumer --namespace ns-wscoz` pod/kafka-htlrnp-kafka-consumer condition met pod_status:NAME READY STATUS RESTARTS AGE kafka-htlrnp-kafka-consumer 1/1 Running 0 5s `kafka-topics.sh --create --topic topic-nhwgd --bootstrap-server kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local:9092` Unable to use a TTY - input is not a terminal or the right kind of file [2025-05-28 03:47:26,642] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:26,759] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:26,961] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:27,264] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:27,768] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:28,674] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:29,681] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:30,789] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:31,895] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:32,801] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:33,907] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:35,113] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:36,320] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:37,427] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:38,434] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:39,340] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:40,546] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:41,451] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:42,657] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:43,663] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:44,769] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:45,774] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:46,980] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:47,986] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:49,092] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:50,098] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:51,003] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:51,908] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:53,013] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:54,118] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:55,323] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:56,328] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) [2025-05-28 03:47:57,228] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local/10.100.181.93:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient) exec return message:Created topic topic-nhwgd. `echo 'kbcli test msg:nhwgd' | kafka-console-producer.sh --topic topic-nhwgd --bootstrap-server kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local:9092` Unable to use a TTY - input is not a terminal or the right kind of file exec return message: `kafka-console-consumer.sh --from-beginning --timeout-ms 10000 --max-messages 1 --topic topic-nhwgd --bootstrap-server kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local:9092` Unable to use a TTY - input is not a terminal or the right kind of file Processed a total of 1 messages exec return msg:kbcli test msg:nhwgd producer msg:[kbcli test msg:nhwgd] equal consumer msg:[kbcli test msg:nhwgd] `kubectl delete pod kafka-htlrnp-kafka-producer kafka-htlrnp-kafka-consumer --force --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "kafka-htlrnp-kafka-producer" force deleted pod "kafka-htlrnp-kafka-consumer" force deleted cluster does not need to check monitor currently check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 test failover podfailure check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kafka-htlrnp" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-kafka-htlrnp.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-kafka-htlrnp created apply test-chaos-mesh-podfailure-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-podfailure-kafka-htlrnp.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Abnormal May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kafka-htlrnp" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover podfailure Success test failover networkdelay check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kafka-htlrnp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-kafka-htlrnp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-kafka-htlrnp created apply test-chaos-mesh-networkdelay-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-networkdelay-kafka-htlrnp.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kafka-htlrnp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover networkdelay Success test failover dnserror check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kafka-htlrnp" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-kafka-htlrnp.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-kafka-htlrnp created apply test-chaos-mesh-dnserror-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-dnserror-kafka-htlrnp.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kafka-htlrnp" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover dnserror Success cluster kafka-broker scale-out check cluster status before ops check cluster status done cluster_status:Running No resources found in kafka-htlrnp namespace. `kbcli cluster scale-out kafka-htlrnp --auto-approve --force=true --components kafka-broker --replicas 1 --namespace ns-wscoz ` OpsRequest kafka-htlrnp-horizontalscaling-dqqt7 created successfully, you can view the progress: kbcli cluster describe-ops kafka-htlrnp-horizontalscaling-dqqt7 -n ns-wscoz check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-horizontalscaling-dqqt7 ns-wscoz HorizontalScaling kafka-htlrnp kafka-broker Running 0/1 May 28,2025 11:56 UTC+0800 check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:56 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 No resources found in kafka-htlrnp namespace. check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-horizontalscaling-dqqt7 ns-wscoz HorizontalScaling kafka-htlrnp kafka-broker Succeed 1/1 May 28,2025 11:56 UTC+0800 check ops status done ops_status:kafka-htlrnp-horizontalscaling-dqqt7 ns-wscoz HorizontalScaling kafka-htlrnp kafka-broker Succeed 1/1 May 28,2025 11:56 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-horizontalscaling-dqqt7 --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-horizontalscaling-dqqt7 patched `kbcli cluster delete-ops --name kafka-htlrnp-horizontalscaling-dqqt7 --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-horizontalscaling-dqqt7 deleted check component kafka-controller exists `kubectl get components -l app.kubernetes.io/instance=kafka-htlrnp,apps.kubeblocks.io/component-name=kafka-controller --namespace ns-wscoz | (grep "kafka-controller" || true )` `kubectl get pvc -l app.kubernetes.io/instance=kafka-htlrnp,apps.kubeblocks.io/component-name=kafka-controller,apps.kubeblocks.io/vct-name=metadata --namespace ns-wscoz ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in kafka-htlrnp namespace. `kbcli cluster volume-expand kafka-htlrnp --auto-approve --force=true --components kafka-controller --volume-claim-templates metadata --storage 4Gi --namespace ns-wscoz ` OpsRequest kafka-htlrnp-volumeexpansion-wvp6l created successfully, you can view the progress: kbcli cluster describe-ops kafka-htlrnp-volumeexpansion-wvp6l -n ns-wscoz check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-volumeexpansion-wvp6l ns-wscoz VolumeExpansion kafka-htlrnp kafka-controller Running 0/3 May 28,2025 11:57 UTC+0800 check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:56 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 No resources found in kafka-htlrnp namespace. check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-volumeexpansion-wvp6l ns-wscoz VolumeExpansion kafka-htlrnp kafka-controller Succeed 3/3 May 28,2025 11:57 UTC+0800 check ops status done ops_status:kafka-htlrnp-volumeexpansion-wvp6l ns-wscoz VolumeExpansion kafka-htlrnp kafka-controller Succeed 3/3 May 28,2025 11:57 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-volumeexpansion-wvp6l --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-volumeexpansion-wvp6l patched `kbcli cluster delete-ops --name kafka-htlrnp-volumeexpansion-wvp6l --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-volumeexpansion-wvp6l deleted test failover networkduplicate check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kafka-htlrnp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-kafka-htlrnp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-kafka-htlrnp created apply test-chaos-mesh-networkduplicate-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-networkduplicate-kafka-htlrnp.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:47 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 11:39 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 11:40 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 11:56 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 11:36 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 11:37 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 11:41 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kafka-htlrnp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover networkduplicate Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart kafka-htlrnp --auto-approve --force=true --namespace ns-wscoz ` OpsRequest kafka-htlrnp-restart-v7vml created successfully, you can view the progress: kbcli cluster describe-ops kafka-htlrnp-restart-v7vml -n ns-wscoz check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-restart-v7vml ns-wscoz Restart kafka-htlrnp kafka-controller,kafka-broker,kafka-exporter Running 0/8 May 28,2025 12:01 UTC+0800 check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:03 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:04 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-restart-v7vml ns-wscoz Restart kafka-htlrnp kafka-controller,kafka-broker,kafka-exporter Succeed 8/8 May 28,2025 12:01 UTC+0800 check ops status done ops_status:kafka-htlrnp-restart-v7vml ns-wscoz Restart kafka-htlrnp kafka-controller,kafka-broker,kafka-exporter Succeed 8/8 May 28,2025 12:01 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-restart-v7vml --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-restart-v7vml patched `kbcli cluster delete-ops --name kafka-htlrnp-restart-v7vml --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-restart-v7vml deleted test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-kafka-htlrnp" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-kafka-htlrnp --namespace ns-wscoz ` Error from server (NotFound): pods "test-db-client-connectionstress-kafka-htlrnp" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-kafka-htlrnp" not found `kubectl get secrets -l app.kubernetes.io/instance=kafka-htlrnp` No resources found in ns-wscoz namespace. Not found cluster secret DB_USERNAME:;DB_PASSWORD:;DB_PORT:9092;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-kafka-htlrnp namespace: ns-wscoz spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "kafka-htlrnp-kafka-broker-advertised-listener-0.ns-wscoz.svc.cluster.local" - "--user" - "" - "--password" - "" - "--port" - "9092" - "--database" - "" - "--dbtype" - "kafka" - "--test" - "connectionstress" - "--connections" - "10000" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-kafka-htlrnp.yaml` pod/test-db-client-connectionstress-kafka-htlrnp created apply test-db-client-connectionstress-kafka-htlrnp.yaml Success `rm -rf test-db-client-connectionstress-kafka-htlrnp.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 63s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 69s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 75s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 80s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 86s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 92s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 98s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 104s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 109s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 115s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m1s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m7s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m13s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m18s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m24s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m36s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m42s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m48s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m53s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 2m59s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 1/1 Running 0 3m46s check pod test-db-client-connectionstress-kafka-htlrnp status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-kafka-htlrnp 0/1 Completed 0 3m51s check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:03 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:04 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-kafka-htlrnp --namespace ns-wscoz ` pod/test-db-client-connectionstress-kafka-htlrnp patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-kafka-htlrnp" force deleted check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover connectionstress Success cluster rebulid instances apiVersion: operations.kubeblocks.io/v1alpha1 kind: OpsRequest metadata: generateName: kafka-htlrnp-rebuildinstance- namespace: ns-wscoz spec: type: RebuildInstance clusterName: kafka-htlrnp force: true rebuildFrom: - componentName: kafka-broker instances: - name: kafka-htlrnp-kafka-broker-0 inPlace: true check cluster status before ops check cluster status done cluster_status:Running `kubectl create -f test_ops_cluster_kafka-htlrnp.yaml` opsrequest.operations.kubeblocks.io/kafka-htlrnp-rebuildinstance-jbz6w created create test_ops_cluster_kafka-htlrnp.yaml Success `rm -rf test_ops_cluster_kafka-htlrnp.yaml` check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Running 0/1 May 28,2025 12:11 UTC+0800 check ops status done ops_status:kafka-htlrnp-rebuildinstance-jbz6w ns-wscoz RebuildInstance kafka-htlrnp kafka-broker Succeed 1/1 May 28,2025 12:11 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-rebuildinstance-jbz6w --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-rebuildinstance-jbz6w patched `kbcli cluster delete-ops --name kafka-htlrnp-rebuildinstance-jbz6w --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-rebuildinstance-jbz6w deleted check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:11 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:04 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 test failover networklossover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kafka-htlrnp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-kafka-htlrnp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-kafka-htlrnp created apply test-chaos-mesh-networklossover-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-networklossover-kafka-htlrnp.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:11 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:04 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kafka-htlrnp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover networklossover Success test failover networkcorruptover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kafka-htlrnp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-kafka-htlrnp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-kafka-htlrnp created apply test-chaos-mesh-networkcorruptover-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-kafka-htlrnp.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-9-41.us-west-2.compute.internal/172.31.9.41 May 28,2025 12:11 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:04 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kafka-htlrnp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover networkcorruptover Success test failover drainnode check node drain check node drain success kubectl get pod kafka-htlrnp-kafka-broker-0 --namespace ns-wscoz -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-9-41.us-west-2.compute.internal success check if multiple pods are on the same node kubectl get pod kafka-htlrnp-kafka-broker-1 --namespace ns-wscoz -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-12-169.us-west-2.compute.internal success kubectl get pod kafka-htlrnp-kafka-broker-2 --namespace ns-wscoz -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-8-216.us-west-2.compute.internal success kubectl get pod kafka-htlrnp-kafka-broker-3 --namespace ns-wscoz -o jsonpath='***.spec.nodeName***' get node name:ip-172-31-5-153.us-west-2.compute.internal success kubectl drain ip-172-31-9-41.us-west-2.compute.internal --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/ip-172-31-9-41.us-west-2.compute.internal cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-xxzjp, kube-system/aws-node-wtk7m, kube-system/ebs-csi-node-zq4zk, kube-system/kube-proxy-8px6m evicting pod ns-wscoz/kafka-htlrnp-kafka-broker-0 evicting pod ns-dyjue/mysql-jjolcb-mysql-0 pod/kafka-htlrnp-kafka-broker-0 evicted pod/mysql-jjolcb-mysql-0 evicted node/ip-172-31-9-41.us-west-2.compute.internal drained kubectl uncordon ip-172-31-9-41.us-west-2.compute.internal node/ip-172-31-9-41.us-west-2.compute.internal uncordoned check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:17 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:05 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 500m / 500m 512Mi / 512Mi metadata:4Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:04 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover drainnode Success check component kafka-controller exists `kubectl get components -l app.kubernetes.io/instance=kafka-htlrnp,apps.kubeblocks.io/component-name=kafka-controller --namespace ns-wscoz | (grep "kafka-controller" || true )` check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale kafka-htlrnp --auto-approve --force=true --components kafka-controller --cpu 600m --memory 0.6Gi --namespace ns-wscoz ` OpsRequest kafka-htlrnp-verticalscaling-5j4dz created successfully, you can view the progress: kbcli cluster describe-ops kafka-htlrnp-verticalscaling-5j4dz -n ns-wscoz check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-verticalscaling-5j4dz ns-wscoz VerticalScaling kafka-htlrnp kafka-controller Running 0/3 May 28,2025 12:18 UTC+0800 check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:17 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:20 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:19 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:18 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-verticalscaling-5j4dz ns-wscoz VerticalScaling kafka-htlrnp kafka-controller Succeed 3/3 May 28,2025 12:18 UTC+0800 check ops status done ops_status:kafka-htlrnp-verticalscaling-5j4dz ns-wscoz VerticalScaling kafka-htlrnp kafka-controller Succeed 3/3 May 28,2025 12:18 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-verticalscaling-5j4dz --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-verticalscaling-5j4dz patched `kbcli cluster delete-ops --name kafka-htlrnp-verticalscaling-5j4dz --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-verticalscaling-5j4dz deleted test failover networkpartition check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kafka-htlrnp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 action: partition mode: all target: mode: all selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-kafka-htlrnp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-kafka-htlrnp created apply test-chaos-mesh-networkpartition-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-networkpartition-kafka-htlrnp.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:17 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:20 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:19 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:18 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kafka-htlrnp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover networkpartition Success test failover timeoffset check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kafka-htlrnp" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-kafka-htlrnp.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-kafka-htlrnp created apply test-chaos-mesh-timeoffset-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-timeoffset-kafka-htlrnp.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:17 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:20 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:19 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:18 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kafka-htlrnp" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover timeoffset Success test failover kill1 check node drain check node drain success `kill 1` Defaulted container "kafka" out of: kafka, jmx-exporter Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:17 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:20 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:19 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:18 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover kill1 Success test failover oom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kafka-htlrnp" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-kafka-htlrnp.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-kafka-htlrnp created apply test-chaos-mesh-oom-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-oom-kafka-htlrnp.yaml` check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:17 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:02 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:01 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:20 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:19 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:18 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:06 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kafka-htlrnp" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover oom Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop kafka-htlrnp --auto-approve --force=true --namespace ns-wscoz ` OpsRequest kafka-htlrnp-stop-qxsv2 created successfully, you can view the progress: kbcli cluster describe-ops kafka-htlrnp-stop-qxsv2 -n ns-wscoz check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-stop-qxsv2 ns-wscoz Stop kafka-htlrnp kafka-broker,kafka-controller,kafka-exporter Running 0/8 May 28,2025 12:27 UTC+0800 check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Stopping May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-stop-qxsv2 ns-wscoz Stop kafka-htlrnp kafka-broker,kafka-controller,kafka-exporter Succeed 8/8 May 28,2025 12:27 UTC+0800 check ops status done ops_status:kafka-htlrnp-stop-qxsv2 ns-wscoz Stop kafka-htlrnp kafka-broker,kafka-controller,kafka-exporter Succeed 8/8 May 28,2025 12:27 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-stop-qxsv2 --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-stop-qxsv2 patched `kbcli cluster delete-ops --name kafka-htlrnp-stop-qxsv2 --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-stop-qxsv2 deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start kafka-htlrnp --force=true --namespace ns-wscoz ` OpsRequest kafka-htlrnp-start-62k5b created successfully, you can view the progress: kbcli cluster describe-ops kafka-htlrnp-start-62k5b -n ns-wscoz check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-start-62k5b ns-wscoz Start kafka-htlrnp kafka-broker,kafka-controller,kafka-exporter Running 0/8 May 28,2025 12:29 UTC+0800 check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:29 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:29 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:32 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:34 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 check ops status `kbcli cluster list-ops kafka-htlrnp --status all --namespace ns-wscoz ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME kafka-htlrnp-start-62k5b ns-wscoz Start kafka-htlrnp kafka-broker,kafka-controller,kafka-exporter Succeed 8/8 May 28,2025 12:29 UTC+0800 check ops status done ops_status:kafka-htlrnp-start-62k5b ns-wscoz Start kafka-htlrnp kafka-broker,kafka-controller,kafka-exporter Succeed 8/8 May 28,2025 12:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests.operations kafka-htlrnp-start-62k5b --namespace ns-wscoz ` opsrequest.operations.kubeblocks.io/kafka-htlrnp-start-62k5b patched `kbcli cluster delete-ops --name kafka-htlrnp-start-62k5b --force --auto-approve --namespace ns-wscoz ` OpsRequest kafka-htlrnp-start-62k5b deleted test failover delete pod:kafka-htlrnp-kafka-broker-0 `kubectl delete pod kafka-htlrnp-kafka-broker-0 --namespace ns-wscoz ` pod "kafka-htlrnp-kafka-broker-0" deleted check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:34 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:29 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:32 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:34 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover Success test failover networkbandwidthover check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kafka-htlrnp" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-kafka-htlrnp.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-kafka-htlrnp created apply test-chaos-mesh-networkbandwidthover-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-kafka-htlrnp.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Updating May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:34 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:29 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:32 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:34 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kafka-htlrnp" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover networkbandwidthover Success test failover dnsrandom check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kafka-htlrnp" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kafka-htlrnp" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-kafka-htlrnp namespace: ns-wscoz spec: selector: namespaces: - ns-wscoz labelSelectors: apps.kubeblocks.io/pod-name: kafka-htlrnp-kafka-broker-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-kafka-htlrnp.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-kafka-htlrnp created apply test-chaos-mesh-dnsrandom-kafka-htlrnp.yaml Success `rm -rf test-chaos-mesh-dnsrandom-kafka-htlrnp.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka DoNotTerminate Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:34 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:29 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:32 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:34 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-kafka-htlrnp --namespace ns-wscoz ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kafka-htlrnp" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-kafka-htlrnp" not found check failover pod name failover pod name:kafka-htlrnp-kafka-broker-0 failover dnsrandom Success cluster list-logs `kbcli cluster list-logs kafka-htlrnp --namespace ns-wscoz ` No log files found. Error from server (NotFound): pods "kafka-htlrnp-kafka-broker-0" not found cluster logs `kbcli cluster logs kafka-htlrnp --tail 30 --namespace ns-wscoz ` Defaulted container "kafka" out of: kafka, jmx-exporter [2025-05-28 04:38:03,916] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-26 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,916] INFO [GroupCoordinator 100]: Elected as the group coordinator for partition 8 in epoch 31 (kafka.coordinator.group.GroupCoordinator) [2025-05-28 04:38:03,916] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-8 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,916] INFO [GroupCoordinator 100]: Elected as the group coordinator for partition 40 in epoch 31 (kafka.coordinator.group.GroupCoordinator) [2025-05-28 04:38:03,916] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-40 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,916] INFO [GroupCoordinator 100]: Elected as the group coordinator for partition 38 in epoch 31 (kafka.coordinator.group.GroupCoordinator) [2025-05-28 04:38:03,916] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-38 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,916] INFO [GroupCoordinator 100]: Elected as the group coordinator for partition 35 in epoch 31 (kafka.coordinator.group.GroupCoordinator) [2025-05-28 04:38:03,916] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-35 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,916] INFO [GroupCoordinator 100]: Elected as the group coordinator for partition 4 in epoch 31 (kafka.coordinator.group.GroupCoordinator) [2025-05-28 04:38:03,917] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-4 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,917] INFO [GroupCoordinator 100]: Elected as the group coordinator for partition 1 in epoch 31 (kafka.coordinator.group.GroupCoordinator) [2025-05-28 04:38:03,917] INFO [GroupMetadataManager brokerId=100] Scheduling loading of offsets and group metadata from __consumer_offsets-1 for epoch 31 (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,920] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-48 in 73 milliseconds for epoch 31, of which 1 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,921] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-13 in 8 milliseconds for epoch 31, of which 8 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,922] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-45 in 8 milliseconds for epoch 31, of which 7 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,922] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-10 in 8 milliseconds for epoch 31, of which 8 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,923] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-42 in 9 milliseconds for epoch 31, of which 9 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,923] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-23 in 9 milliseconds for epoch 31, of which 9 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,924] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-17 in 9 milliseconds for epoch 31, of which 9 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,924] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-18 in 9 milliseconds for epoch 31, of which 9 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,925] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-30 in 10 milliseconds for epoch 31, of which 10 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,925] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-27 in 10 milliseconds for epoch 31, of which 10 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,926] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-26 in 10 milliseconds for epoch 31, of which 10 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,926] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-8 in 10 milliseconds for epoch 31, of which 10 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,927] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-40 in 11 milliseconds for epoch 31, of which 11 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,927] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-38 in 11 milliseconds for epoch 31, of which 11 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,928] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-35 in 12 milliseconds for epoch 31, of which 11 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,928] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-4 in 11 milliseconds for epoch 31, of which 11 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) [2025-05-28 04:38:03,928] INFO [GroupMetadataManager brokerId=100] Finished loading offsets and group metadata from __consumer_offsets-1 in 11 milliseconds for epoch 31, of which 11 milliseconds was spent in the scheduler. (kafka.coordinator.group.GroupMetadataManager) cluster update terminationPolicy WipeOut `kbcli cluster update kafka-htlrnp --termination-policy=WipeOut --namespace ns-wscoz ` cluster.apps.kubeblocks.io/kafka-htlrnp updated check cluster status `kbcli cluster list kafka-htlrnp --show-labels --namespace ns-wscoz ` NAME NAMESPACE CLUSTER-DEFINITION TERMINATION-POLICY STATUS CREATED-TIME LABELS kafka-htlrnp ns-wscoz kafka WipeOut Running May 28,2025 11:36 UTC+0800 app.kubernetes.io/instance=kafka-htlrnp,clusterdefinition.kubeblocks.io/name=kafka check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances kafka-htlrnp --namespace ns-wscoz ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME kafka-htlrnp-kafka-broker-0 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-12-169.us-west-2.compute.internal/172.31.12.169 May 28,2025 12:34 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-1 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-67.us-west-2.compute.internal/172.31.4.67 May 28,2025 12:29 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-2 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-broker-3 ns-wscoz kafka-htlrnp kafka-broker Running us-west-2a 500m / 500m 512Mi / 512Mi data:1Gi ip-172-31-4-134.us-west-2.compute.internal/172.31.4.134 May 28,2025 12:31 UTC+0800 metadata:1Gi kafka-htlrnp-kafka-controller-0 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-8-216.us-west-2.compute.internal/172.31.8.216 May 28,2025 12:32 UTC+0800 kafka-htlrnp-kafka-controller-1 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-153.us-west-2.compute.internal/172.31.5.153 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-controller-2 ns-wscoz kafka-htlrnp kafka-controller Running us-west-2a 600m / 600m 644245094400m / 644245094400m metadata:4Gi ip-172-31-5-43.us-west-2.compute.internal/172.31.5.43 May 28,2025 12:33 UTC+0800 kafka-htlrnp-kafka-exporter-0 ns-wscoz kafka-htlrnp kafka-exporter Running us-west-2a 500m / 500m 512Mi / 512Mi ip-172-31-4-184.us-west-2.compute.internal/172.31.4.184 May 28,2025 12:34 UTC+0800 check pod status done connect unsupported engine type: kafka kafka-broker-1.0.0-alpha.0 delete cluster kafka-htlrnp `kbcli cluster delete kafka-htlrnp --auto-approve --namespace ns-wscoz ` Cluster kafka-htlrnp deleted pod_info:kafka-htlrnp-kafka-broker-0 2/2 Running 1 (3m49s ago) 5m59s kafka-htlrnp-kafka-broker-1 2/2 Terminating 0 10m kafka-htlrnp-kafka-broker-2 2/2 Terminating 0 9m40s kafka-htlrnp-kafka-broker-3 2/2 Terminating 0 8m53s kafka-htlrnp-kafka-controller-0 2/2 Running 0 8m7s kafka-htlrnp-kafka-controller-1 2/2 Running 0 7m33s kafka-htlrnp-kafka-controller-2 2/2 Running 0 7m kafka-htlrnp-kafka-exporter-0 1/1 Running 0 6m19s pod_info:kafka-htlrnp-kafka-exporter-0 1/1 Terminating 0 6m40s No resources found in ns-wscoz namespace. delete cluster pod done No resources found in ns-wscoz namespace. check cluster resource non-exist OK: pvc No resources found in ns-wscoz namespace. delete cluster done No resources found in ns-wscoz namespace. No resources found in ns-wscoz namespace. No resources found in ns-wscoz namespace. Kafka Test Suite All Done! --------------------------------------Kafka (Topology = separated_monitor Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ComponentDefinition=kafka-broker-1.0.0-alpha.0;ComponentVersion=kafka-broker;ServiceVersion=3.3.2;]|[Description=Create a cluster with the specified component definition kafka-broker-1.0.0-alpha.0 and component version kafka-broker and service version 3.3.2] [PASSED]|[Failover]|[HA=Full CPU;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Connect]|[Endpoints=true]|[Description=Connect to the cluster] [PASSED]|[Update]|[Monitor=true]|[Description=Update the cluster Monitor enable] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Failover]|[HA=Network Delay;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[Failover]|[HA=DNS Error;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[HorizontalScaling Out]|[ComponentName=kafka-broker;Topology=separated]|[Description=HorizontalScaling Out the cluster specify component kafka-broker] [PASSED]|[VolumeExpansion]|[ComponentName=kafka-controller]|[Description=VolumeExpansion the cluster specify component kafka-controller] [PASSED]|[Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=kafka-broker]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[RebuildInstance]|[ComponentName=kafka-broker]|[Description=Rebuild the cluster instance specify component kafka-broker] [PASSED]|[Failover]|[HA=Network Loss Failover;Durations=2m;ComponentName=kafka-broker]|[Description=] [PASSED]|[Failover]|[HA=Network Corrupt Failover;Durations=2m;ComponentName=kafka-broker]|[Description=] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=kafka-broker]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[VerticalScaling]|[ComponentName=kafka-controller]|[Description=VerticalScaling the cluster specify component kafka-controller] [PASSED]|[Failover]|[HA=Network Partition;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Time Offset;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=kafka-broker]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[Failover]|[HA=Delete Pod;ComponentName=kafka-broker]|[Description=Simulates conditions where pods terminating forced/graceful thereby testing deployment sanity (replica availability & uninterrupted service) and recovery workflow of the application.] [PASSED]|[Failover]|[HA=Network Bandwidth Failover;Durations=2m;ComponentName=kafka-broker]|[Description=] [PASSED]|[Failover]|[HA=DNS Random;Durations=2m;ComponentName=kafka-broker]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]