source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-rurgs ` `kubectl create namespace ns-rurgs` namespace/ns-rurgs created create namespace ns-rurgs done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.4-beta.1` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 48.9M 0 --:--:-- --:--:-- --:--:-- 48.9M kbcli installed successfully. Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.4-beta.1 done Kubernetes: v1.32.5-eks-5d4a308 KubeBlocks: 0.9.4 kbcli: 0.9.4-beta.1 WARNING: version difference between kbcli (0.9.4-beta.1) and kubeblocks (0.9.4) Kubernetes Env: v1.32.5-eks-5d4a308 POD_RESOURCES: No resources found found default storage class: gp3 kubeblocks version is:0.9.4 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:ob-bundle set component version No resources found no component version found not found component version set replicas first:3|2 CLUSTER_TOPOLOGY: Not found topology in cluster definition ob-ce LIMIT_CPU:3 LIMIT_MEMORY:4 storage size: 50 No resources found in ns-rurgs namespace. termination_policy:DoNotTerminate create 3 replica DoNotTerminate ob-ce cluster check cluster version check cluster definition check component definition set component definition by component version no component definitions found apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: obce-fhizqq namespace: ns-rurgs annotations: "kubeblocks.io/extra-env": '***"TENANT_NAME":"tenant3","ZONE_COUNT":"3","OB_CLUSTERS_COUNT":"1","TENANT_CPU":"2","TENANT_MEMORY":"2G","TENANT_DISK":"10G"***' spec: terminationPolicy: DoNotTerminate componentSpecs: - name: ob-bundle componentDefRef: ob-ce componentDef: ob-ce replicas: 3 resources: requests: cpu: 3000m memory: 4Gi limits: cpu: 3000m memory: 4Gi volumeClaimTemplates: - name: data-file spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "50Gi" - name: data-log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "50Gi" - name: log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "10Gi" - name: workdir spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "100Mi" `kubectl apply -f test_create_obce-fhizqq.yaml` cluster.apps.kubeblocks.io/obce-fhizqq created apply test_create_obce-fhizqq.yaml Success `rm -rf test_create_obce-fhizqq.yaml` check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Jun 19,2025 18:14 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase There is no password in Type: 16. describe cluster `kbcli cluster describe obce-fhizqq --namespace ns-rurgs ` Name: obce-fhizqq Created Time: Jun 19,2025 18:14 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-rurgs Running DoNotTerminate Endpoints: COMPONENT MODE INTERNAL EXTERNAL ob-bundle ReadWrite obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local:2881 obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local:2882 obce-fhizqq-ob-bundle-ordinal-1.ns-rurgs.svc.cluster.local:2881 obce-fhizqq-ob-bundle-ordinal-1.ns-rurgs.svc.cluster.local:2882 obce-fhizqq-ob-bundle-ordinal-2.ns-rurgs.svc.cluster.local:2881 obce-fhizqq-ob-bundle-ordinal-2.ns-rurgs.svc.cluster.local:2882 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME ob-bundle obce-fhizqq-ob-bundle-0 Running us-west-2a ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 ob-bundle obce-fhizqq-ob-bundle-1 Running us-west-2a ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 ob-bundle obce-fhizqq-ob-bundle-2 Running us-west-2a ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS ob-bundle false 3 / 3 4Gi / 4Gi data-file:50Gi kb-default-sc data-log:50Gi kb-default-sc log:10Gi kb-default-sc workdir:100Mi kb-default-sc Images: COMPONENT TYPE IMAGE ob-bundle ob-ce docker.io/apecloud/oceanbase:4.3.0.1-100000242024032211 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-rurgs obce-fhizqq `kbcli cluster label obce-fhizqq app.kubernetes.io/instance- --namespace ns-rurgs ` label "app.kubernetes.io/instance" not found. `kbcli cluster label obce-fhizqq app.kubernetes.io/instance=obce-fhizqq --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq label cluster app.kubernetes.io/instance=obce-fhizqq Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=obce-fhizqq --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label obce-fhizqq case.name=kbcli.test2 --overwrite --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label obce-fhizqq case.name- --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"SHOW PARAMETERS LIKE '%syslog_level%'\G;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash ` Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file ERROR 1146 (42S02) at line 1: Table 'oceanbase.__all_virtual_tenant_parameter_stat' doesn't exist command terminated with exit code 1 check connect cluster... Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file ERROR 4012 (HY000) at line 1: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. command terminated with exit code 1 check connect cluster... Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file ERROR 4012 (HY000) at line 1: Timeout, query has reached the maximum query timeout: 10000000(us), maybe you can adjust the session variable ob_query_timeout or query_timeout hint, and try again. command terminated with exit code 1 check connect cluster... Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file *************************** 1. row *************************** zone: zone2 svr_type: observer svr_ip: 10.100.6.168 svr_port: 2882 name: syslog_level data_type: STRING value: DEBUG info: specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. section: OBSERVER scope: CLUSTER source: DEFAULT edit_level: DYNAMIC_EFFECTIVE *************************** 2. row *************************** zone: zone0 svr_type: observer svr_ip: 10.100.148.142 svr_port: 2882 name: syslog_level data_type: STRING value: DEBUG info: specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. section: OBSERVER scope: CLUSTER source: DEFAULT edit_level: DYNAMIC_EFFECTIVE *************************** 3. row *************************** zone: zone1 svr_type: observer svr_ip: 10.100.246.144 svr_port: 2882 name: syslog_level data_type: STRING value: DEBUG info: specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. section: OBSERVER scope: CLUSTER source: DEFAULT edit_level: DYNAMIC_EFFECTIVE connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-obce-fhizqq" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-obce-fhizqq --namespace ns-rurgs ` Error from server (NotFound): pods "test-db-client-executionloop-obce-fhizqq" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-obce-fhizqq" not found `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local" - "--user" - "root" - "--password" - "" - "--port" - "2881" - "--dbtype" - "oceanbase" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-obce-fhizqq.yaml` pod/test-db-client-executionloop-obce-fhizqq created apply test-db-client-executionloop-obce-fhizqq.yaml Success `rm -rf test-db-client-executionloop-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 52s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 63s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 69s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 75s check pod test-db-client-executionloop-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 0/1 Completed 0 81s check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Running Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done --host obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local --user root --password --port 2881 --dbtype oceanbase --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:18:34.177 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); Execution loop start:INSERT INTO executions_loop.executions_loop_table (value) VALUES ('executions_loop_test_1'); [ 60s ] executions total: 1 successful: 1 failed: 0 disconnect: 0 Test Result: Total Executions: 1 Successful Executions: 1 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: oceanbase Host: obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local Port: 2881 Database: Table: User: root Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 1 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-obce-fhizqq --namespace ns-rurgs ` pod/test-db-client-executionloop-obce-fhizqq patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-obce-fhizqq" force deleted get oceanbase compatibility mode COMPATIBILITY_MOD:mysql check tenant role `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"SELECT TENANT_NAME,TENANT_TYPE,CREATE_TIME,STATUS,TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME like 'tenant%';\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-0 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file `echo " echo \"SELECT TENANT_NAME,TENANT_TYPE,CREATE_TIME,STATUS,TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME like 'tenant%';\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file `echo " echo \"SELECT TENANT_NAME,TENANT_TYPE,CREATE_TIME,STATUS,TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME like 'tenant%';\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file check tenant role [PRIMARY][PRIMARY][PRIMARY] Success TENANT_NAME:tenant3 `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); INSERT INTO tmp_table (value) VALUES ('vfhol');\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file add consistent data vfhol Success skip cluster SwitchOver: not support `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase apiVersion: v1 kind: Pod metadata: name: benchtest-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local,user:root,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-obce-fhizqq.yaml` pod/benchtest-obce-fhizqq created apply benchtest-obce-fhizqq.yaml Success `rm -rf benchtest-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 23s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 47s check pod benchtest-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 Completed 0 52s `kubectl logs benchtest-obce-fhizqq --tail 30 --namespace ns-rurgs ` [ 7s ] thds: 4 tps: 39.01 qps: 766.24 (r/w/o: 533.17/155.05/78.02) lat (ms,99%): 161.51 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 41.00 qps: 821.92 (r/w/o: 575.94/163.98/81.99) lat (ms,99%): 132.49 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 42.01 qps: 840.18 (r/w/o: 587.12/169.04/84.02) lat (ms,99%): 155.80 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 42.00 qps: 841.95 (r/w/o: 589.97/167.99/84.00) lat (ms,99%): 114.72 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 5586 write: 1596 other: 798 total: 7980 transactions: 399 (39.63 per sec.) queries: 7980 (792.55 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0672s total number of events: 399 Latency (ms): min: 65.87 avg: 100.75 max: 221.09 99th percentile: 196.89 sum: 40197.52 Threads fairness: events (avg/stddev): 99.7500/0.43 execution time (avg/stddev): 10.0494/0.02 `kubectl delete pod benchtest-obce-fhizqq --force --namespace ns-rurgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-obce-fhizqq" force deleted cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose obce-fhizqq --auto-approve --force=true --type internet --enable true --components ob-bundle --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-6sb2j created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-expose-6sb2j -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-6sb2j ns-rurgs Expose obce-fhizqq ob-bundle Running 0/1 Jun 19,2025 18:22 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Running Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-6sb2j ns-rurgs Expose obce-fhizqq ob-bundle Succeed 1/1 Jun 19,2025 18:22 UTC+0800 check ops status done ops_status:obce-fhizqq-expose-6sb2j ns-rurgs Expose obce-fhizqq ob-bundle Succeed 1/1 Jun 19,2025 18:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-expose-6sb2j --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-expose-6sb2j patched `kbcli cluster delete-ops --name obce-fhizqq-expose-6sb2j --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-6sb2j deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value vfhol check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check db_client batch data Success No resources found in ns-rurgs namespace. `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase apiVersion: v1 kind: Pod metadata: name: benchtest-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com,user:root,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-obce-fhizqq.yaml` pod/benchtest-obce-fhizqq created apply benchtest-obce-fhizqq.yaml Success `rm -rf benchtest-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 0s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 23s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 52s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 63s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 69s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 75s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 81s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 86s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 92s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 98s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 104s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 110s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 115s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m7s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m13s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m24s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m30s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m36s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m42s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m47s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m53s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 2m59s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m11s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m16s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m22s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m28s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m34s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m40s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m46s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m51s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 3m57s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m3s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m9s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m14s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m20s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m26s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m32s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m38s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m43s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m49s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 4m55s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m24s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m30s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m35s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m41s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5m47s check pod benchtest-obce-fhizqq status timeout --------------------------------------get pod benchtest-obce-fhizqq yaml-------------------------------------- `kubectl get pod benchtest-obce-fhizqq -o yaml --namespace ns-rurgs ` apiVersion: v1 kind: Pod metadata: annotations: kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"v1","kind":"Pod","metadata":***"annotations":***,"name":"benchtest-obce-fhizqq","namespace":"ns-rurgs"***,"spec":***"containers":[***"env":[***"name":"TYPE","value":"2"***,***"name":"FLAG","value":"0"***,***"name":"CONFIGS","value":"mode:all,driver:mysql,host:a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com,user:root,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write"***],"image":"docker.io/apecloud/customsuites:latest","imagePullPolicy":"IfNotPresent","name":"test-sysbench"***],"restartPolicy":"Never"*** creationTimestamp: "2025-06-19T10:23:43Z" name: benchtest-obce-fhizqq namespace: ns-rurgs resourceVersion: "31475" uid: 19fe0237-5ef4-4f0c-8af1-407ec3eb3e7d spec: containers: - env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: mode:all,driver:mysql,host:a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com,user:root,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write image: docker.io/apecloud/customsuites:latest imagePullPolicy: IfNotPresent name: test-sysbench resources: *** terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-q7dxs readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true nodeName: ip-172-31-14-102.us-west-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: Never schedulerName: default-scheduler securityContext: *** serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - name: kube-api-access-q7dxs projected: defaultMode: 420 sources: - serviceAccountToken: expirationSeconds: 3607 path: token - configMap: items: - key: ca.crt path: ca.crt name: kube-root-ca.crt - downwardAPI: items: - fieldRef: apiVersion: v1 fieldPath: metadata.namespace path: namespace status: conditions: - lastProbeTime: null lastTransitionTime: "2025-06-19T10:23:44Z" status: "True" type: PodReadyToStartContainers - lastProbeTime: null lastTransitionTime: "2025-06-19T10:23:43Z" status: "True" type: Initialized - lastProbeTime: null lastTransitionTime: "2025-06-19T10:23:44Z" status: "True" type: Ready - lastProbeTime: null lastTransitionTime: "2025-06-19T10:23:44Z" status: "True" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2025-06-19T10:23:43Z" status: "True" type: PodScheduled containerStatuses: - containerID: containerd://a4fed89b10838f72b1975f3c3e2d5ed202bd355bbb840c1ae80cbe6dd4eecda7 image: docker.io/apecloud/customsuites:latest imageID: docker.io/apecloud/customsuites@sha256:97ac1738b48fc3e19dc817715c0257037a7fff84f18e0afbe6e5aa572e8a0cd1 lastState: *** name: test-sysbench ready: true restartCount: 0 started: true state: running: startedAt: "2025-06-19T10:23:43Z" volumeMounts: - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: kube-api-access-q7dxs readOnly: true recursiveReadOnly: Disabled hostIP: 172.31.14.102 hostIPs: - ip: 172.31.14.102 phase: Running podIP: 172.31.11.160 podIPs: - ip: 172.31.11.160 qosClass: BestEffort startTime: "2025-06-19T10:23:43Z" ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe pod benchtest-obce-fhizqq-------------------------------------- `kubectl describe pod benchtest-obce-fhizqq --namespace ns-rurgs ` Name: benchtest-obce-fhizqq Namespace: ns-rurgs Priority: 0 Service Account: default Node: ip-172-31-14-102.us-west-2.compute.internal/172.31.14.102 Start Time: Thu, 19 Jun 2025 18:23:43 +0800 Labels: Annotations: Status: Running IP: 172.31.11.160 IPs: IP: 172.31.11.160 Containers: test-sysbench: Container ID: containerd://a4fed89b10838f72b1975f3c3e2d5ed202bd355bbb840c1ae80cbe6dd4eecda7 Image: docker.io/apecloud/customsuites:latest Image ID: docker.io/apecloud/customsuites@sha256:97ac1738b48fc3e19dc817715c0257037a7fff84f18e0afbe6e5aa572e8a0cd1 Port: Host Port: State: Running Started: Thu, 19 Jun 2025 18:23:43 +0800 Ready: True Restart Count: 0 Environment: TYPE: 2 FLAG: 0 CONFIGS: mode:all,driver:mysql,host:a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com,user:root,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write Mounts: /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-q7dxs (ro) Conditions: Type Status PodReadyToStartContainers True Initialized True Ready True ContainersReady True PodScheduled True Volumes: kube-api-access-q7dxs: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 3607 ConfigMapName: kube-root-ca.crt ConfigMapOptional: DownwardAPI: true QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300s Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 5m49s default-scheduler Successfully assigned ns-rurgs/benchtest-obce-fhizqq to ip-172-31-14-102.us-west-2.compute.internal Normal Pulled 5m49s kubelet Container image "docker.io/apecloud/customsuites:latest" already present on machine Normal Created 5m49s kubelet Created container: test-sysbench Normal Started 5m49s kubelet Started container test-sysbench ------------------------------------------------------------------------------------------------------------------ --------------------------------------pod benchtest-obce-fhizqq-------------------------------------- `kubectl logs benchtest-obce-fhizqq --namespace ns-rurgs --tail 500` run sysbench check mysql database exists mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect fail:(2003, "Can't connect to MySQL server on 'a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com' ([Errno -2] Name or service not known)") mysql host a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com connect success! database benchtest exists /usr/bin/sysbench cd ./sysbench/src/lua;sysbench --db-driver=mysql --mysql-host=a24ddd14020534bcf8da820b9198933c-e241145cd7855a14.elb.us-west-2.amazonaws.com --mysql-port=2881 --mysql-user=root --mysql-password= --mysql-db=benchtest --table_size=1000 --tables=5 --time=10 --threads=4 oltp_read_write cleanup sysbench 1.0.20 (using system LuaJIT 2.1.0-beta3) ------------------------------------------------------------------------------------------------------------------ `kubectl delete pod benchtest-obce-fhizqq --force --namespace ns-rurgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-obce-fhizqq" force deleted `kubectl get pvc -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle,apps.kubeblocks.io/vct-name=data-file --namespace ns-rurgs ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in obce-fhizqq namespace. `kbcli cluster volume-expand obce-fhizqq --auto-approve --force=true --components ob-bundle --volume-claim-templates data-file --storage 54Gi --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-jbgz5 created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-volumeexpansion-jbgz5 -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-jbgz5 ns-rurgs VolumeExpansion obce-fhizqq ob-bundle Running -/- Jun 19,2025 18:29 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Updating Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done No resources found in obce-fhizqq namespace. check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-jbgz5 ns-rurgs VolumeExpansion obce-fhizqq ob-bundle Succeed 3/3 Jun 19,2025 18:29 UTC+0800 check ops status done ops_status:obce-fhizqq-volumeexpansion-jbgz5 ns-rurgs VolumeExpansion obce-fhizqq ob-bundle Succeed 3/3 Jun 19,2025 18:29 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-volumeexpansion-jbgz5 --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-volumeexpansion-jbgz5 patched `kbcli cluster delete-ops --name obce-fhizqq-volumeexpansion-jbgz5 --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-jbgz5 deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value vfhol check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check db_client batch data Success 10 skip cluster hscale test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-obce-fhizqq" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-obce-fhizqq --namespace ns-rurgs ` Error from server (NotFound): pods "test-db-client-connectionstress-obce-fhizqq" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-obce-fhizqq" not found `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local" - "--user" - "root" - "--password" - "" - "--port" - "2881" - "--database" - "oceanbase" - "--dbtype" - "oceanbase" - "--test" - "connectionstress" - "--connections" - "5000" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-obce-fhizqq.yaml` pod/test-db-client-connectionstress-obce-fhizqq created apply test-db-client-connectionstress-obce-fhizqq.yaml Success `rm -rf test-db-client-connectionstress-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 16s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 39s check pod test-db-client-connectionstress-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 0/1 Completed 0 45s check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Running Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done --host obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local --user root --password --port 2881 --database oceanbase --dbtype oceanbase --test connectionstress --connections 5000 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:31:43.911 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Test Result: Created 5000 connections Connection Information: Database Type: oceanbase Host: obce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local Port: 2881 Database: oceanbase Table: User: root Org: Access Mode: mysql Test Type: connectionstress Connection Count: 5000 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-obce-fhizqq --namespace ns-rurgs ` pod/test-db-client-connectionstress-obce-fhizqq patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-obce-fhizqq" force deleted check failover pod name failover pod name:obce-fhizqq-ob-bundle-2 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value vfhol check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check db_client batch data Success cluster configure component_tmp: ob-bundle check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure obce-fhizqq --auto-approve --force=true --set system_memory=2G --components ob-bundle --config-spec ob-ce-config --config-file oceanbase.conf --namespace ns-rurgs ` Will updated configure file meta: ConfigSpec: ob-ce-config ConfigFile: oceanbase.conf ComponentName: ob-bundle ClusterName: obce-fhizqq OpsRequest obce-fhizqq-reconfiguring-g9bq6 created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-reconfiguring-g9bq6 -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-reconfiguring-g9bq6 ns-rurgs Reconfiguring obce-fhizqq ob-bundle Running -/- Jun 19,2025 18:32 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Running Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-reconfiguring-g9bq6 ns-rurgs Reconfiguring obce-fhizqq ob-bundle Succeed 3/3 Jun 19,2025 18:32 UTC+0800 check ops status done ops_status:obce-fhizqq-reconfiguring-g9bq6 ns-rurgs Reconfiguring obce-fhizqq ob-bundle Succeed 3/3 Jun 19,2025 18:32 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-reconfiguring-g9bq6 --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-reconfiguring-g9bq6 patched `kbcli cluster delete-ops --name obce-fhizqq-reconfiguring-g9bq6 --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-reconfiguring-g9bq6 deleted check config variables `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 2G configure:[system_memory] result actual:[2G] equal expected:[2G] `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value vfhol check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check db_client batch data Success skip cluster stop: not support `kubectl get pvc -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle,apps.kubeblocks.io/vct-name=data-log --namespace ns-rurgs ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in obce-fhizqq namespace. `kbcli cluster volume-expand obce-fhizqq --auto-approve --force=true --components ob-bundle --volume-claim-templates data-log --storage 52Gi --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-rwwpr created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-volumeexpansion-rwwpr -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-rwwpr ns-rurgs VolumeExpansion obce-fhizqq ob-bundle Running 0/3 Jun 19,2025 18:33 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs DoNotTerminate Updating Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done No resources found in obce-fhizqq namespace. check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-rwwpr ns-rurgs VolumeExpansion obce-fhizqq ob-bundle Succeed 3/3 Jun 19,2025 18:33 UTC+0800 check ops status done ops_status:obce-fhizqq-volumeexpansion-rwwpr ns-rurgs VolumeExpansion obce-fhizqq ob-bundle Succeed 3/3 Jun 19,2025 18:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-volumeexpansion-rwwpr --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-volumeexpansion-rwwpr patched `kbcli cluster delete-ops --name obce-fhizqq-volumeexpansion-rwwpr --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-rwwpr deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value vfhol check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check db_client batch data Success skip cluster vscale: not support skip cluster restart: not support cluster update terminationPolicy WipeOut `kbcli cluster update obce-fhizqq --termination-policy=WipeOut --namespace ns-rurgs ` cluster.apps.kubeblocks.io/obce-fhizqq updated check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs WipeOut Running Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done cluster list-logs `kbcli cluster list-logs obce-fhizqq --namespace ns-rurgs ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update obce-fhizqq --enable-all-logs=true --namespace ns-rurgs Error from server (NotFound): pods "obce-fhizqq-ob-bundle-2" not found cluster logs `kbcli cluster logs obce-fhizqq --tail 30 --namespace ns-rurgs ` Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) CREATE_TIME: 2025-06-19 18:18:03.796536 MODIFY_TIME: 2025-06-19 18:18:09.181065 BUILD_VERSION: 4.3.0.1_100000242024032211-0193a343bc60b4699ec47792c3fc4ce166a182f9(Mar 22 2024 13:07:59) LAST_OFFLINE_TIME: NULL /scripts/bootstrap.sh: line 322: [: bundle: integer expression expected create resource unit and pool for tenant tenant3 create tenant tenant3 alter system archive log check tenant tenant3 exists count(*) 1 *************************** 1. row *************************** TENANT_NAME: sys TENANT_TYPE: SYS TENANT_ROLE: PRIMARY SWITCHOVER_STATUS: NORMAL *************************** 2. row *************************** TENANT_NAME: META$1002 TENANT_TYPE: META TENANT_ROLE: PRIMARY SWITCHOVER_STATUS: NORMAL *************************** 3. row *************************** TENANT_NAME: tenant3 TENANT_TYPE: USER TENANT_ROLE: PRIMARY SWITCHOVER_STATUS: NORMAL svr_ip_list: 10.100.148.142 create user rep_user for tenant tenant3 Bootstrap successfully Server started successfully cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose obce-fhizqq --auto-approve --force=true --type internet --enable false --components ob-bundle --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-hqgsf created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-expose-hqgsf -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-hqgsf ns-rurgs Expose obce-fhizqq ob-bundle Running 0/1 Jun 19,2025 18:34 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs WipeOut Running Jun 19,2025 18:14 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-8-181.us-west-2.compute.internal/172.31.8.181 Jun 19,2025 18:14 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:15 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-2 ns-rurgs obce-fhizqq ob-bundle Running us-west-2a 3200m / 4 4352Mi / 5Gi data-file:54Gi ip-172-31-3-195.us-west-2.compute.internal/172.31.3.195 Jun 19,2025 18:16 UTC+0800 data-log:52Gi log:10Gi workdir:100Mi check pod status done check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-hqgsf ns-rurgs Expose obce-fhizqq ob-bundle Succeed 1/1 Jun 19,2025 18:34 UTC+0800 check ops status done ops_status:obce-fhizqq-expose-hqgsf ns-rurgs Expose obce-fhizqq ob-bundle Succeed 1/1 Jun 19,2025 18:34 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-expose-hqgsf --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-expose-hqgsf patched `kbcli cluster delete-ops --name obce-fhizqq-expose-hqgsf --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-hqgsf deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value vfhol check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-2 --namespace ns-rurgs -- bash` check db_client batch data Success delete cluster obce-fhizqq `kbcli cluster delete obce-fhizqq --auto-approve --namespace ns-rurgs ` Cluster obce-fhizqq deleted pod_info:obce-fhizqq-ob-bundle-0 3/3 Terminating 0 20m obce-fhizqq-ob-bundle-1 3/3 Terminating 0 19m obce-fhizqq-ob-bundle-2 3/3 Terminating 0 18m pod_info:obce-fhizqq-ob-bundle-0 3/3 Terminating 0 20m obce-fhizqq-ob-bundle-1 3/3 Terminating 0 19m obce-fhizqq-ob-bundle-2 3/3 Terminating 0 19m pod_info:obce-fhizqq-ob-bundle-0 3/3 Terminating 0 20m obce-fhizqq-ob-bundle-1 3/3 Terminating 0 20m obce-fhizqq-ob-bundle-2 3/3 Terminating 0 19m No resources found in ns-rurgs namespace. delete cluster pod done No resources found in ns-rurgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-rurgs namespace. delete cluster done No resources found in ns-rurgs namespace. No resources found in ns-rurgs namespace. No resources found in ns-rurgs namespace. --------------------------------------OceanBase CE (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=ob-ce;ClusterVersion=ob-ce-4.3.0.1-100000242024032211;]|[Description=Create a cluster with the specified cluster definition ob-ce and cluster version ob-ce-4.3.0.1-100000242024032211] [PASSED]|[Connect]|[ComponentName=ob-bundle]|[Description=Connect to the cluster] [PASSED]|[Check Role]|[Tenant=tenant3]|[Description=Check the cluster tenant role] [PASSED]|[AddData]|[Values=vfhol]|[Description=Add data to the cluster] [PASSED]|[Bench]|[ComponentName=ob-bundle]|[Description=Bench the cluster service with ob-bundle component] [PASSED]|[Expose]|[Enable=true;TYPE=internet;ComponentName=ob-bundle]|[Description=Expose Enable the internet service with ob-bundle component] [FAILED]|[Bench]|[HostType=LB;ComponentName=ob-bundle]|[Description=Bench the cluster LB service with ob-bundle component] [PASSED]|[VolumeExpansion]|[ComponentName=ob-bundle;ComponentVolume=data-file]|[Description=VolumeExpansion the cluster specify component ob-bundle and volume data-file] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=ob-bundle]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Reconfiguring]|[ComponentName=ob-bundle;system_memory=2G]|[Description=Reconfiguring the cluster specify component ob-bundle set system_memory=2G] [PASSED]|[VolumeExpansion]|[ComponentName=ob-bundle;ComponentVolume=data-log]|[Description=VolumeExpansion the cluster specify component ob-bundle and volume data-log] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Expose]|[Disable=true;TYPE=internet;ComponentName=ob-bundle]|[Description=Expose Disable the internet service with ob-bundle component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] CLUSTER_TOPOLOGY: Not found topology in cluster definition ob-ce LIMIT_CPU:3 LIMIT_MEMORY:4 storage size: 50 No resources found in ns-rurgs namespace. termination_policy:Halt create 2 replica Halt ob-ce cluster check cluster version check cluster definition check component definition set component definition by component version no component definitions found apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: obce-fhizqq namespace: ns-rurgs annotations: "kubeblocks.io/extra-env": '***"TENANT_NAME":"tenant2","ZONE_COUNT":"1","OB_CLUSTERS_COUNT":"2","TENANT_CPU":"2","TENANT_MEMORY":"2G","TENANT_DISK":"10G"***' spec: terminationPolicy: Halt componentSpecs: - name: ob-bundle-0 componentDefRef: ob-ce-repl componentDef: ob-ce-repl replicas: 1 resources: requests: cpu: 3000m memory: 4Gi limits: cpu: 3000m memory: 4Gi volumeClaimTemplates: - name: data-file spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "50Gi" - name: data-log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "50Gi" - name: log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "10Gi" - name: workdir spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "100Mi" - name: ob-bundle-1 componentDefRef: ob-ce-repl componentDef: ob-ce-repl replicas: 1 resources: requests: cpu: 3000m memory: 4Gi limits: cpu: 3000m memory: 4Gi volumeClaimTemplates: - name: data-file spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "50Gi" - name: data-log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "50Gi" - name: log spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "10Gi" - name: workdir spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: "100Mi" `kubectl apply -f test_create_obce-fhizqq.yaml` cluster.apps.kubeblocks.io/obce-fhizqq created apply test_create_obce-fhizqq.yaml Success `rm -rf test_create_obce-fhizqq.yaml` check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Jun 19,2025 18:36 UTC+0800 cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase There is no password in Type: 16. describe cluster `kbcli cluster describe obce-fhizqq --namespace ns-rurgs ` Name: obce-fhizqq Created Time: Jun 19,2025 18:36 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-rurgs Running Halt Endpoints: COMPONENT MODE INTERNAL EXTERNAL ob-bundle-0 ReadWrite obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local:2881 obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local:2882 ob-bundle-1 ReadWrite obce-fhizqq-ob-bundle-1-ordinal-0.ns-rurgs.svc.cluster.local:2881 obce-fhizqq-ob-bundle-1-ordinal-0.ns-rurgs.svc.cluster.local:2882 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME ob-bundle-0 obce-fhizqq-ob-bundle-0-0 primary Running us-west-2a ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 ob-bundle-1 obce-fhizqq-ob-bundle-1-0 standby Running us-west-2a ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS ob-bundle-0 false 3 / 3 4Gi / 4Gi data-file:50Gi kb-default-sc data-log:50Gi kb-default-sc log:10Gi kb-default-sc workdir:100Mi kb-default-sc ob-bundle-1 false 3 / 3 4Gi / 4Gi data-file:50Gi kb-default-sc data-log:50Gi kb-default-sc log:10Gi kb-default-sc workdir:100Mi kb-default-sc Images: COMPONENT TYPE IMAGE ob-bundle-0 ob-ce-repl docker.io/apecloud/oceanbase:4.3.0.1-100000242024032211 ob-bundle-1 ob-ce-repl docker.io/apecloud/oceanbase:4.3.0.1-100000242024032211 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-rurgs obce-fhizqq `kbcli cluster label obce-fhizqq app.kubernetes.io/instance- --namespace ns-rurgs ` label "app.kubernetes.io/instance" not found. `kbcli cluster label obce-fhizqq app.kubernetes.io/instance=obce-fhizqq --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq label cluster app.kubernetes.io/instance=obce-fhizqq Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=obce-fhizqq --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq case.name=kbcli.test1 label cluster case.name=kbcli.test1 Success `kbcli cluster label obce-fhizqq case.name=kbcli.test2 --overwrite --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq case.name=kbcli.test2 label cluster case.name=kbcli.test2 Success `kbcli cluster label obce-fhizqq case.name- --namespace ns-rurgs ` `kbcli cluster label obce-fhizqq --list --namespace ns-rurgs ` NAME NAMESPACE LABELS obce-fhizqq ns-rurgs app.kubernetes.io/instance=obce-fhizqq delete cluster label case.name Success cluster connect `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"SHOW PARAMETERS LIKE '%syslog_level%'\G;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-0-0 --namespace ns-rurgs -- bash ` Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file *************************** 1. row *************************** zone: zone0 svr_type: observer svr_ip: 10.100.164.13 svr_port: 2882 name: syslog_level data_type: STRING value: DEBUG info: specifies the current level of logging. There are DEBUG, TRACE, WDIAG, EDIAG, INFO, WARN, ERROR, seven different log levels. section: OBSERVER scope: CLUSTER source: DEFAULT edit_level: DYNAMIC_EFFECTIVE connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-obce-fhizqq" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-obce-fhizqq --namespace ns-rurgs ` Error from server (NotFound): pods "test-db-client-executionloop-obce-fhizqq" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-obce-fhizqq" not found `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase No resources found in ns-rurgs namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local" - "--user" - "root@tenant2" - "--password" - "" - "--port" - "2881" - "--dbtype" - "oceanbase" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-obce-fhizqq.yaml` pod/test-db-client-executionloop-obce-fhizqq created apply test-db-client-executionloop-obce-fhizqq.yaml Success `rm -rf test-db-client-executionloop-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 45s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 57s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 1/1 Running 0 63s check pod test-db-client-executionloop-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-obce-fhizqq 0/1 Completed 0 69s check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done --host obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local --user root@tenant2 --password --port 2881 --dbtype oceanbase --test executionloop --duration 60 --interval 1 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:42:28.372 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Execution loop start: create databases executions_loop CREATE DATABASE IF NOT EXISTS executions_loop; drop table executions_loop_table DROP TABLE IF EXISTS executions_loop.executions_loop_table; create table executions_loop_table CREATE TABLE IF NOT EXISTS executions_loop.executions_loop_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); Execution loop start:INSERT INTO executions_loop.executions_loop_table (value) VALUES ('executions_loop_test_1'); [ 1s ] executions total: 127 successful: 127 failed: 0 disconnect: 0 [ 2s ] executions total: 356 successful: 356 failed: 0 disconnect: 0 [ 3s ] executions total: 585 successful: 585 failed: 0 disconnect: 0 [ 4s ] executions total: 857 successful: 857 failed: 0 disconnect: 0 [ 5s ] executions total: 1127 successful: 1127 failed: 0 disconnect: 0 [ 6s ] executions total: 1384 successful: 1384 failed: 0 disconnect: 0 [ 7s ] executions total: 1679 successful: 1679 failed: 0 disconnect: 0 [ 8s ] executions total: 1958 successful: 1958 failed: 0 disconnect: 0 [ 9s ] executions total: 2235 successful: 2235 failed: 0 disconnect: 0 [ 10s ] executions total: 2514 successful: 2514 failed: 0 disconnect: 0 [ 11s ] executions total: 2774 successful: 2774 failed: 0 disconnect: 0 [ 12s ] executions total: 3030 successful: 3030 failed: 0 disconnect: 0 [ 13s ] executions total: 3314 successful: 3314 failed: 0 disconnect: 0 [ 14s ] executions total: 3587 successful: 3587 failed: 0 disconnect: 0 [ 15s ] executions total: 3828 successful: 3828 failed: 0 disconnect: 0 [ 16s ] executions total: 4106 successful: 4106 failed: 0 disconnect: 0 [ 17s ] executions total: 4375 successful: 4375 failed: 0 disconnect: 0 [ 18s ] executions total: 4652 successful: 4652 failed: 0 disconnect: 0 [ 19s ] executions total: 4924 successful: 4924 failed: 0 disconnect: 0 [ 20s ] executions total: 5188 successful: 5188 failed: 0 disconnect: 0 [ 21s ] executions total: 5449 successful: 5449 failed: 0 disconnect: 0 [ 22s ] executions total: 5696 successful: 5696 failed: 0 disconnect: 0 [ 23s ] executions total: 5991 successful: 5991 failed: 0 disconnect: 0 [ 24s ] executions total: 6258 successful: 6258 failed: 0 disconnect: 0 [ 25s ] executions total: 6512 successful: 6512 failed: 0 disconnect: 0 [ 26s ] executions total: 6771 successful: 6771 failed: 0 disconnect: 0 [ 27s ] executions total: 7042 successful: 7042 failed: 0 disconnect: 0 [ 28s ] executions total: 7329 successful: 7329 failed: 0 disconnect: 0 [ 29s ] executions total: 7609 successful: 7609 failed: 0 disconnect: 0 [ 30s ] executions total: 7892 successful: 7892 failed: 0 disconnect: 0 [ 31s ] executions total: 8190 successful: 8190 failed: 0 disconnect: 0 [ 32s ] executions total: 8452 successful: 8452 failed: 0 disconnect: 0 [ 33s ] executions total: 8706 successful: 8706 failed: 0 disconnect: 0 [ 34s ] executions total: 8966 successful: 8966 failed: 0 disconnect: 0 [ 35s ] executions total: 9221 successful: 9221 failed: 0 disconnect: 0 [ 36s ] executions total: 9464 successful: 9464 failed: 0 disconnect: 0 [ 37s ] executions total: 9742 successful: 9742 failed: 0 disconnect: 0 [ 38s ] executions total: 10017 successful: 10017 failed: 0 disconnect: 0 [ 39s ] executions total: 10286 successful: 10286 failed: 0 disconnect: 0 [ 40s ] executions total: 10564 successful: 10564 failed: 0 disconnect: 0 [ 41s ] executions total: 10817 successful: 10817 failed: 0 disconnect: 0 [ 42s ] executions total: 11044 successful: 11044 failed: 0 disconnect: 0 [ 43s ] executions total: 11324 successful: 11324 failed: 0 disconnect: 0 [ 44s ] executions total: 11587 successful: 11587 failed: 0 disconnect: 0 [ 45s ] executions total: 11880 successful: 11880 failed: 0 disconnect: 0 [ 46s ] executions total: 12157 successful: 12157 failed: 0 disconnect: 0 [ 47s ] executions total: 12432 successful: 12432 failed: 0 disconnect: 0 [ 48s ] executions total: 12691 successful: 12691 failed: 0 disconnect: 0 [ 49s ] executions total: 12938 successful: 12938 failed: 0 disconnect: 0 [ 50s ] executions total: 13209 successful: 13209 failed: 0 disconnect: 0 [ 51s ] executions total: 13476 successful: 13476 failed: 0 disconnect: 0 [ 52s ] executions total: 13724 successful: 13724 failed: 0 disconnect: 0 [ 53s ] executions total: 13994 successful: 13994 failed: 0 disconnect: 0 [ 54s ] executions total: 14269 successful: 14269 failed: 0 disconnect: 0 [ 55s ] executions total: 14539 successful: 14539 failed: 0 disconnect: 0 [ 56s ] executions total: 14818 successful: 14818 failed: 0 disconnect: 0 [ 57s ] executions total: 15092 successful: 15092 failed: 0 disconnect: 0 [ 58s ] executions total: 15381 successful: 15381 failed: 0 disconnect: 0 [ 59s ] executions total: 15677 successful: 15677 failed: 0 disconnect: 0 [ 60s ] executions total: 15930 successful: 15930 failed: 0 disconnect: 0 Test Result: Total Executions: 15930 Successful Executions: 15930 Failed Executions: 0 Disconnection Counts: 0 Connection Information: Database Type: oceanbase Host: obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local Port: 2881 Database: Table: User: root@tenant2 Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 15930 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-obce-fhizqq --namespace ns-rurgs ` pod/test-db-client-executionloop-obce-fhizqq patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-obce-fhizqq" force deleted get oceanbase compatibility mode COMPATIBILITY_MOD:mysql check tenant role `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"SELECT TENANT_NAME,TENANT_TYPE,CREATE_TIME,STATUS,TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME like 'tenant%';\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-0-0 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file `echo " echo \"SELECT TENANT_NAME,TENANT_TYPE,CREATE_TIME,STATUS,TENANT_ROLE FROM oceanbase.DBA_OB_TENANTS WHERE TENANT_NAME like 'tenant%';\" | mysql -hobce-fhizqq-ob-bundle-1-ordinal-0.ns-rurgs.svc.cluster.local -uroot -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file check tenant role [PRIMARY][STANDBY] Success TENANT_NAME:tenant2 `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; DROP TABLE IF EXISTS tmp_table; CREATE TABLE IF NOT EXISTS tmp_table (id INT PRIMARY KEY AUTO_INCREMENT, value VARCHAR(255)); INSERT INTO tmp_table (value) VALUES ('yvuri');\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-0-0 --namespace ns-rurgs -- bash` Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file add consistent data yvuri Success 10 skip cluster HorizontalScaling out skip cluster vscale: not support skip cluster stop: not support check component ob-bundle-0 exists `kubectl get components -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle-0 --namespace ns-rurgs | (grep "ob-bundle-0" || true )` `kubectl get pvc -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle-0,apps.kubeblocks.io/vct-name=data-file --namespace ns-rurgs ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in obce-fhizqq namespace. `kbcli cluster volume-expand obce-fhizqq --auto-approve --force=true --components ob-bundle-0 --volume-claim-templates data-file --storage 52Gi --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-zx6mx created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-volumeexpansion-zx6mx -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-zx6mx ns-rurgs VolumeExpansion obce-fhizqq ob-bundle-0 Running 0/1 Jun 19,2025 18:46 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done No resources found in obce-fhizqq namespace. check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-zx6mx ns-rurgs VolumeExpansion obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 18:46 UTC+0800 check ops status done ops_status:obce-fhizqq-volumeexpansion-zx6mx ns-rurgs VolumeExpansion obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 18:46 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-volumeexpansion-zx6mx --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-volumeexpansion-zx6mx patched `kbcli cluster delete-ops --name obce-fhizqq-volumeexpansion-zx6mx --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-zx6mx deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value yvuri check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check db_client batch data Success cluster configure check component ob-bundle-0 exists `kubectl get components -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle-0 --namespace ns-rurgs | (grep "ob-bundle-0" || true )` check component ob-bundle-1 exists `kubectl get components -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle-1 --namespace ns-rurgs | (grep "ob-bundle-1" || true )` component_tmp: ob-bundle-0,ob-bundle-1 check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure obce-fhizqq --auto-approve --force=true --set system_memory=2G --components ob-bundle-0,ob-bundle-1 --config-spec ob-ce-config --config-file oceanbase.conf --namespace ns-rurgs ` Will updated configure file meta: ConfigSpec: ob-ce-config ConfigFile: oceanbase.conf ComponentName: ob-bundle-0 ClusterName: obce-fhizqq OpsRequest obce-fhizqq-reconfiguring-r25tl created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-reconfiguring-r25tl -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-reconfiguring-r25tl ns-rurgs Reconfiguring obce-fhizqq ob-bundle-0,ob-bundle-1 Running -/- Jun 19,2025 18:49 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-reconfiguring-r25tl ns-rurgs Reconfiguring obce-fhizqq ob-bundle-0,ob-bundle-1 Succeed 2/2 Jun 19,2025 18:49 UTC+0800 check ops status done ops_status:obce-fhizqq-reconfiguring-r25tl ns-rurgs Reconfiguring obce-fhizqq ob-bundle-0,ob-bundle-1 Succeed 2/2 Jun 19,2025 18:49 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-reconfiguring-r25tl --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-reconfiguring-r25tl patched `kbcli cluster delete-ops --name obce-fhizqq-reconfiguring-r25tl --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-reconfiguring-r25tl deleted check config variables `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 2G configure:[system_memory] result actual:[2G] equal expected:[2G] `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value yvuri check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check db_client batch data Success check component ob-bundle-0 exists `kubectl get components -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle-0 --namespace ns-rurgs | (grep "ob-bundle-0" || true )` `kubectl get pvc -l app.kubernetes.io/instance=obce-fhizqq,apps.kubeblocks.io/component-name=ob-bundle-0,apps.kubeblocks.io/vct-name=data-log --namespace ns-rurgs ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in obce-fhizqq namespace. `kbcli cluster volume-expand obce-fhizqq --auto-approve --force=true --components ob-bundle-0 --volume-claim-templates data-log --storage 55Gi --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-6xxb4 created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-volumeexpansion-6xxb4 -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-6xxb4 ns-rurgs VolumeExpansion obce-fhizqq ob-bundle-0 Running 0/1 Jun 19,2025 18:52 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Updating Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:55Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done No resources found in obce-fhizqq namespace. check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-volumeexpansion-6xxb4 ns-rurgs VolumeExpansion obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 18:52 UTC+0800 check ops status done ops_status:obce-fhizqq-volumeexpansion-6xxb4 ns-rurgs VolumeExpansion obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 18:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-volumeexpansion-6xxb4 --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-volumeexpansion-6xxb4 patched `kbcli cluster delete-ops --name obce-fhizqq-volumeexpansion-6xxb4 --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-volumeexpansion-6xxb4 deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value yvuri check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check db_client batch data Success skip cluster restart: not support test failover connectionstress check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-obce-fhizqq" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-obce-fhizqq --namespace ns-rurgs ` Error from server (NotFound): pods "test-db-client-connectionstress-obce-fhizqq" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-obce-fhizqq" not found `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase No resources found in ns-rurgs namespace. apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local" - "--user" - "root@tenant2" - "--password" - "" - "--port" - "2881" - "--database" - "oceanbase" - "--dbtype" - "oceanbase" - "--test" - "connectionstress" - "--connections" - "5000" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-obce-fhizqq.yaml` pod/test-db-client-connectionstress-obce-fhizqq created apply test-db-client-connectionstress-obce-fhizqq.yaml Success `rm -rf test-db-client-connectionstress-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 0/1 ContainerCreating 0 11s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 17s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 22s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 28s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 34s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 1/1 Running 0 45s check pod test-db-client-connectionstress-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-obce-fhizqq 0/1 Completed 0 51s check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:55Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done --host obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local --user root@tenant2 --password --port 2881 --database oceanbase --dbtype oceanbase --test connectionstress --connections 5000 --duration 60 SLF4J(I): Connected with provider of type [ch.qos.logback.classic.spi.LogbackServiceProvider] 10:56:57.852 [main] DEBUG com.clickhouse.jdbc.ClickHouseDriver -- ClickHouse Driver 0.0.0.0(JDBC: 0.0.0.0) registered Test Result: Created 5000 connections Connection Information: Database Type: oceanbase Host: obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local Port: 2881 Database: oceanbase Table: User: root@tenant2 Org: Access Mode: mysql Test Type: connectionstress Connection Count: 5000 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-obce-fhizqq --namespace ns-rurgs ` pod/test-db-client-connectionstress-obce-fhizqq patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-obce-fhizqq" force deleted check failover pod name failover pod name:obce-fhizqq-ob-bundle-0-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value yvuri check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check db_client batch data Success skip cluster SwitchOver: not support `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase apiVersion: v1 kind: Pod metadata: name: benchtest-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:obce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local,user:root@tenant2,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-obce-fhizqq.yaml` pod/benchtest-obce-fhizqq created apply benchtest-obce-fhizqq.yaml Success `rm -rf benchtest-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 6s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 12s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 18s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 23s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 29s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 41s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 46s check pod benchtest-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 Completed 0 52s `kubectl logs benchtest-obce-fhizqq --tail 30 --namespace ns-rurgs ` [ 7s ] thds: 4 tps: 157.42 qps: 3153.32 (r/w/o: 2208.82/629.66/314.83) lat (ms,99%): 43.39 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 151.93 qps: 3055.66 (r/w/o: 2142.06/609.73/303.87) lat (ms,99%): 38.94 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 151.04 qps: 3022.87 (r/w/o: 2112.61/608.18/302.09) lat (ms,99%): 33.12 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 130.70 qps: 2608.98 (r/w/o: 1826.79/520.80/261.40) lat (ms,99%): 65.65 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 18452 write: 5272 other: 2636 total: 26360 transactions: 1318 (131.48 per sec.) queries: 26360 (2629.66 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0226s total number of events: 1318 Latency (ms): min: 18.20 avg: 30.38 max: 93.23 99th percentile: 69.29 sum: 40038.88 Threads fairness: events (avg/stddev): 329.5000/2.29 execution time (avg/stddev): 10.0097/0.01 `kubectl delete pod benchtest-obce-fhizqq --force --namespace ns-rurgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-obce-fhizqq" force deleted cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose obce-fhizqq --auto-approve --force=true --type internet --enable true --components ob-bundle-0 --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-vdg9h created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-expose-vdg9h -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-vdg9h ns-rurgs Expose obce-fhizqq ob-bundle-0 Running 0/1 Jun 19,2025 19:02 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs Halt Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:55Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-vdg9h ns-rurgs Expose obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 19:02 UTC+0800 check ops status done ops_status:obce-fhizqq-expose-vdg9h ns-rurgs Expose obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 19:02 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-expose-vdg9h --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-expose-vdg9h patched `kbcli cluster delete-ops --name obce-fhizqq-expose-vdg9h --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-vdg9h deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value yvuri check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check db_client batch data Success No resources found in ns-rurgs namespace. `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase apiVersion: v1 kind: Pod metadata: name: benchtest-obce-fhizqq namespace: ns-rurgs spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:mysql,host:afea2197157df4a99a6c2faa08f5204a-e81c974b1bf1d7c4.elb.us-west-2.amazonaws.com,user:root@tenant2,password:,port:2881,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-obce-fhizqq.yaml` pod/benchtest-obce-fhizqq created apply benchtest-obce-fhizqq.yaml Success `rm -rf benchtest-obce-fhizqq.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 1/1 Running 0 11s check pod benchtest-obce-fhizqq status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-obce-fhizqq 0/1 Completed 0 17s `kubectl logs benchtest-obce-fhizqq --tail 30 --namespace ns-rurgs ` [ 7s ] thds: 4 tps: 135.44 qps: 2723.88 (r/w/o: 1906.21/546.78/270.88) lat (ms,99%): 42.61 err/s: 0.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 129.01 qps: 2567.26 (r/w/o: 1795.18/514.05/258.03) lat (ms,99%): 44.17 err/s: 0.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 134.01 qps: 2668.11 (r/w/o: 1868.08/532.02/268.01) lat (ms,99%): 51.02 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 133.98 qps: 2673.67 (r/w/o: 1866.77/538.93/267.97) lat (ms,99%): 42.61 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 19754 write: 5644 other: 2822 total: 28220 transactions: 1411 (140.56 per sec.) queries: 28220 (2811.18 per sec.) ignored errors: 0 (0.00 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0370s total number of events: 1411 Latency (ms): min: 19.99 avg: 28.40 max: 77.82 99th percentile: 44.98 sum: 40079.40 Threads fairness: events (avg/stddev): 352.7500/14.58 execution time (avg/stddev): 10.0198/0.01 `kubectl delete pod benchtest-obce-fhizqq --force --namespace ns-rurgs ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-obce-fhizqq" force deleted cluster update terminationPolicy WipeOut `kbcli cluster update obce-fhizqq --termination-policy=WipeOut --namespace ns-rurgs ` cluster.apps.kubeblocks.io/obce-fhizqq updated check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs WipeOut Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:55Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done cluster list-logs `kbcli cluster list-logs obce-fhizqq --namespace ns-rurgs ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update obce-fhizqq --enable-all-logs=true --namespace ns-rurgs Error from server (NotFound): pods "obce-fhizqq-ob-bundle-0-0" not found cluster logs `kbcli cluster logs obce-fhizqq --tail 30 --namespace ns-rurgs ` Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) BLOCK_MIGRATE_IN_TIME: NULL CREATE_TIME: 2025-06-19 18:38:16.930975 MODIFY_TIME: 2025-06-19 18:38:22.350437 BUILD_VERSION: 4.3.0.1_100000242024032211-0193a343bc60b4699ec47792c3fc4ce166a182f9(Mar 22 2024 13:07:59) LAST_OFFLINE_TIME: NULL 10.100.180.150 CREATE RESOURCE UNIT IF NOT EXISTS unit_for_tenant2 10.100.180.150 CREATE RESOURCE POOL IF NOT EXISTS pool_for_tenant2 UNIT = 'unit_for_tenant2', UNIT_NUM = 1; create tenant tenant2 10.100.180.150 SET SESSION ob_query_timeout=1000000000; CREATE STANDBY TENANT IF NOT EXISTS tenant2 LOG_RESTORE_SOURCE ='SERVICE=10.100.164.13:2881; USER=rep_user@tenant2 ***' RESOURCE_POOL_LIST=('pool_for_tenant2'); 10.100.180.150 ALTER SYSTEM ARCHIVELOG; check tenant tenant2 exists count(*) 1 *************************** 1. row *************************** TENANT_NAME: sys TENANT_TYPE: SYS TENANT_ROLE: PRIMARY SWITCHOVER_STATUS: NORMAL *************************** 2. row *************************** TENANT_NAME: META$1002 TENANT_TYPE: META TENANT_ROLE: PRIMARY SWITCHOVER_STATUS: NORMAL *************************** 3. row *************************** TENANT_NAME: tenant2 TENANT_TYPE: USER TENANT_ROLE: STANDBY SWITCHOVER_STATUS: NORMAL Bootstrap successfully Server started successfully cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose obce-fhizqq --auto-approve --force=true --type internet --enable false --components ob-bundle-0 --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-sxfxx created successfully, you can view the progress: kbcli cluster describe-ops obce-fhizqq-expose-sxfxx -n ns-rurgs check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-sxfxx ns-rurgs Expose obce-fhizqq ob-bundle-0 Running 0/1 Jun 19,2025 19:09 UTC+0800 check cluster status `kbcli cluster list obce-fhizqq --show-labels --namespace ns-rurgs ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS obce-fhizqq ns-rurgs WipeOut Running Jun 19,2025 18:36 UTC+0800 app.kubernetes.io/instance=obce-fhizqq check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances obce-fhizqq --namespace ns-rurgs ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME obce-fhizqq-ob-bundle-0-0 ns-rurgs obce-fhizqq ob-bundle-0 Running primary us-west-2a 3200m / 4 4352Mi / 5Gi data-file:52Gi ip-172-31-4-34.us-west-2.compute.internal/172.31.4.34 Jun 19,2025 18:36 UTC+0800 data-log:55Gi log:10Gi workdir:100Mi obce-fhizqq-ob-bundle-1-0 ns-rurgs obce-fhizqq ob-bundle-1 Running standby us-west-2a 3200m / 4 4352Mi / 5Gi data-file:50Gi ip-172-31-2-196.us-west-2.compute.internal/172.31.2.196 Jun 19,2025 18:36 UTC+0800 data-log:50Gi log:10Gi workdir:100Mi check pod status done check cluster role No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: No resources found in ns-rurgs namespace. primary: obce-fhizqq-ob-bundle-0-0;secondary: check cluster connect `echo '' | kbcli cluster connect obce-fhizqq --namespace ns-rurgs` check cluster connect done check ops status `kbcli cluster list-ops obce-fhizqq --status all --namespace ns-rurgs ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME obce-fhizqq-expose-sxfxx ns-rurgs Expose obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 19:09 UTC+0800 check ops status done ops_status:obce-fhizqq-expose-sxfxx ns-rurgs Expose obce-fhizqq ob-bundle-0 Succeed 1/1 Jun 19,2025 19:09 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests obce-fhizqq-expose-sxfxx --namespace ns-rurgs ` opsrequest.apps.kubeblocks.io/obce-fhizqq-expose-sxfxx patched `kbcli cluster delete-ops --name obce-fhizqq-expose-sxfxx --force --auto-approve --namespace ns-rurgs ` OpsRequest obce-fhizqq-expose-sxfxx deleted `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase `echo " echo \"use test; SELECT value FROM tmp_table WHERE id = 1;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check data: Defaulted container "observer-container" out of: observer-container, metrics, lorry, config-manager, kb-tools (init) Unable to use a TTY - input is not a terminal or the right kind of file value yvuri check cluster data consistent Success `kubectl get secrets -l app.kubernetes.io/instance=obce-fhizqq` No resources found in ns-rurgs namespace. Not found cluster secret DB_USERNAME:root@tenant2;DB_PASSWORD:;DB_PORT:2881;DB_DATABASE:oceanbase check db_client batch data count `echo " echo \"select count(*) from executions_loop.executions_loop_table;\" | mysql -hobce-fhizqq-ob-bundle-0-ordinal-0.ns-rurgs.svc.cluster.local -uroot@tenant2 -P2881 " | kubectl exec -it obce-fhizqq-ob-bundle-1-0 --namespace ns-rurgs -- bash` check db_client batch data Success delete cluster obce-fhizqq `kbcli cluster delete obce-fhizqq --auto-approve --namespace ns-rurgs ` Cluster obce-fhizqq deleted pod_info:obce-fhizqq-ob-bundle-0-0 4/4 Running 0 36m obce-fhizqq-ob-bundle-1-0 4/4 Running 0 36m pod_info:obce-fhizqq-ob-bundle-0-0 3/4 Terminating 0 36m obce-fhizqq-ob-bundle-1-0 3/4 Terminating 0 36m pod_info:obce-fhizqq-ob-bundle-0-0 3/4 Terminating 0 37m obce-fhizqq-ob-bundle-1-0 3/4 Terminating 0 36m No resources found in ns-rurgs namespace. delete cluster pod done No resources found in ns-rurgs namespace. check cluster resource non-exist OK: pvc No resources found in ns-rurgs namespace. delete cluster done No resources found in ns-rurgs namespace. No resources found in ns-rurgs namespace. No resources found in ns-rurgs namespace. OceanBase CE Test Suite All Done! --------------------------------------OceanBase CE (Topology = Replicas 3) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=ob-ce;ClusterVersion=ob-ce-4.3.0.1-100000242024032211;]|[Description=Create a cluster with the specified cluster definition ob-ce and cluster version ob-ce-4.3.0.1-100000242024032211] [PASSED]|[Connect]|[ComponentName=ob-bundle]|[Description=Connect to the cluster] [PASSED]|[Check Role]|[Tenant=tenant3]|[Description=Check the cluster tenant role] [PASSED]|[AddData]|[Values=vfhol]|[Description=Add data to the cluster] [PASSED]|[Bench]|[ComponentName=ob-bundle]|[Description=Bench the cluster service with ob-bundle component] [PASSED]|[Expose]|[Enable=true;TYPE=internet;ComponentName=ob-bundle]|[Description=Expose Enable the internet service with ob-bundle component] [FAILED]|[Bench]|[HostType=LB;ComponentName=ob-bundle]|[Description=Bench the cluster LB service with ob-bundle component] [PASSED]|[VolumeExpansion]|[ComponentName=ob-bundle;ComponentVolume=data-file]|[Description=VolumeExpansion the cluster specify component ob-bundle and volume data-file] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=ob-bundle]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Reconfiguring]|[ComponentName=ob-bundle;system_memory=2G]|[Description=Reconfiguring the cluster specify component ob-bundle set system_memory=2G] [PASSED]|[VolumeExpansion]|[ComponentName=ob-bundle;ComponentVolume=data-log]|[Description=VolumeExpansion the cluster specify component ob-bundle and volume data-log] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Expose]|[Disable=true;TYPE=internet;ComponentName=ob-bundle]|[Description=Expose Disable the internet service with ob-bundle component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] --------------------------------------OceanBase CE (Topology = Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=ob-ce;ClusterVersion=ob-ce-4.3.0.1-100000242024032211;]|[Description=Create a cluster with the specified cluster definition ob-ce and cluster version ob-ce-4.3.0.1-100000242024032211] [PASSED]|[Connect]|[ComponentName=ob-bundle]|[Description=Connect to the cluster] [PASSED]|[Check Role]|[Tenant=tenant2]|[Description=Check the cluster tenant role] [PASSED]|[AddData]|[Values=yvuri]|[Description=Add data to the cluster] [PASSED]|[VolumeExpansion]|[ComponentName=ob-bundle-0;ComponentVolume=data-file]|[Description=VolumeExpansion the cluster specify component ob-bundle-0 and volume data-file] [PASSED]|[Reconfiguring]|[ComponentName=ob-bundle;system_memory=2G]|[Description=Reconfiguring the cluster specify component ob-bundle set system_memory=2G] [PASSED]|[VolumeExpansion]|[ComponentName=ob-bundle-0;ComponentVolume=data-log]|[Description=VolumeExpansion the cluster specify component ob-bundle-0 and volume data-log] [PASSED]|[Failover]|[HA=Connection Stress;ComponentName=ob-bundle]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[Bench]|[ComponentName=ob-bundle]|[Description=Bench the cluster service with ob-bundle component] [PASSED]|[Expose]|[Enable=true;TYPE=internet;ComponentName=ob-bundle-0]|[Description=Expose Enable the internet service with ob-bundle-0 component] [PASSED]|[Bench]|[HostType=LB;ComponentName=ob-bundle]|[Description=Bench the cluster LB service with ob-bundle component] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Expose]|[Disable=true;TYPE=internet;ComponentName=ob-bundle-0]|[Description=Expose Disable the internet service with ob-bundle-0 component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]