bash test/kbcli/test_kbcli_0.9.sh --type 36 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-lrqxb ` `kubectl create namespace ns-lrqxb` namespace/ns-lrqxb created create namespace ns-lrqxb done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 16 32.1M 16 5338k 0 0 43.9M 0 --:--:-- --:--:-- --:--:-- 43.9M 100 32.1M 100 32.1M 0 0 164M 0 --:--:-- --:--:-- --:--:-- 355M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:oracle set component version set component version:oracle set service versions:12.2.0,23.6.0 set service versions sorted:12.2.0,23.6.0 no cluster version found unsupported component definition REPORT_COUNT 0:0 set replicas first:2,12.2.0 set replicas third:2,12.2.0 set replicas fourth:2,12.2.0 set minimum cmpv service version set minimum cmpv service version replicas:2,12.2.0 REPORT_COUNT:1 CLUSTER_TOPOLOGY:replication topology replication found in cluster definition oracle LIMIT_CPU:4 LIMIT_MEMORY:8 storage size: 20 No resources found in ns-lrqxb namespace. termination_policy:Delete create 2 replica Delete oracle cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition1: oracle-12c by component version:oracle apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: oracle-hzwvnn namespace: ns-lrqxb spec: clusterDefinitionRef: oracle topology: replication terminationPolicy: Delete componentSpecs: - name: oracle replicas: 2 serviceVersion: 12.2.0 monitor: true disableExporter: false resources: requests: cpu: 4000m memory: 8Gi limits: cpu: 4000m memory: 8Gi volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: fra spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: observer replicas: 1 serviceVersion: 12.2.0 resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi `kubectl apply -f test_create_oracle-hzwvnn.yaml` cluster.apps.kubeblocks.io/oracle-hzwvnn created apply test_create_oracle-hzwvnn.yaml Success `rm -rf test_create_oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Sep 01,2025 11:19 UTC+0800 clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check pod oracle-hzwvnn-oracle-0 container_name oracle exist password U970u#h1N76u check pod oracle-hzwvnn-oracle-0 container_name exporter exist password U970u#h1N76u check pod oracle-hzwvnn-oracle-0 container_name lorry exist password U970u#h1N76u check pod oracle-hzwvnn-oracle-0 container_name config-manager exist password U970u#h1N76u No container logs contain secret password. describe cluster `kbcli cluster describe oracle-hzwvnn --namespace ns-lrqxb ` Name: oracle-hzwvnn Created Time: Sep 01,2025 11:19 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-lrqxb oracle Running Delete Endpoints: COMPONENT MODE INTERNAL EXTERNAL oracle ReadWrite oracle-hzwvnn-oracle.ns-lrqxb.svc.cluster.local:1521 oracle-hzwvnn-oracle.ns-lrqxb.svc.cluster.local:5500 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME observer oracle-hzwvnn-observer-0 Running 0 aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle oracle-hzwvnn-oracle-0 primary Running 0 aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 oracle oracle-hzwvnn-oracle-1 secondary Running 0 aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS oracle false 4 / 4 8Gi / 8Gi data:20Gi default fra:20Gi default observer false 500m / 500m 512Mi / 512Mi Images: COMPONENT TYPE IMAGE oracle docker.io/apecloud/oracle:12.2.0.1-ee observer docker.io/apecloud/oracle:12.2.0.1-ee Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-lrqxb oracle-hzwvnn `kbcli cluster label oracle-hzwvnn app.kubernetes.io/instance- --namespace ns-lrqxb ` label "app.kubernetes.io/instance" not found. `kbcli cluster label oracle-hzwvnn app.kubernetes.io/instance=oracle-hzwvnn --namespace ns-lrqxb ` `kbcli cluster label oracle-hzwvnn --list --namespace ns-lrqxb ` NAME NAMESPACE LABELS oracle-hzwvnn ns-lrqxb app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=oracle-hzwvnn Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=oracle-hzwvnn --namespace ns-lrqxb ` `kbcli cluster label oracle-hzwvnn --list --namespace ns-lrqxb ` NAME NAMESPACE LABELS oracle-hzwvnn ns-lrqxb app.kubernetes.io/instance=oracle-hzwvnn case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label oracle-hzwvnn case.name=kbcli.test2 --overwrite --namespace ns-lrqxb ` `kbcli cluster label oracle-hzwvnn --list --namespace ns-lrqxb ` NAME NAMESPACE LABELS oracle-hzwvnn ns-lrqxb app.kubernetes.io/instance=oracle-hzwvnn case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label oracle-hzwvnn case.name- --namespace ns-lrqxb ` `kbcli cluster label oracle-hzwvnn --list --namespace ns-lrqxb ` NAME NAMESPACE LABELS oracle-hzwvnn ns-lrqxb app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect `echo "echo \"select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,STATUS,DATABASE_STATUS,INSTANCE_ROLE,DATABASE_TYPE from v\\$instance;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` Defaulted container "oracle" out of: oracle, exporter, lorry, config-manager, oracle-init-container (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file INSTANCE_NUMBER INSTANCE_NAME --------------- ---------------- HOST_NAME STATUS ---------------------------------------------------------------- ------------ DATABASE_STATUS INSTANCE_ROLE DATABASE_TYPE ----------------- ------------------ --------------- 1 ORCLCDB oracle-hzwvnn-oracle-0 OPEN ACTIVE PRIMARY_INSTANCE SINGLE connect cluster Success insert batch data by db client Error from server (NotFound): pods "test-db-client-executionloop-oracle-hzwvnn" not found DB_CLIENT_BATCH_DATA_COUNT: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): pods "test-db-client-executionloop-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-executionloop-oracle-hzwvnn" not found `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-executionloop-oracle-hzwvnn namespace: ns-lrqxb spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "oracle-hzwvnn-oracle.ns-lrqxb.svc.cluster.local" - "--user" - "sys" - "--password" - "U970u#h1N76u" - "--port" - "1521" - "--dbtype" - "oracle" - "--test" - "executionloop" - "--duration" - "60" - "--interval" - "1" restartPolicy: Never `kubectl apply -f test-db-client-executionloop-oracle-hzwvnn.yaml` pod/test-db-client-executionloop-oracle-hzwvnn created apply test-db-client-executionloop-oracle-hzwvnn.yaml Success `rm -rf test-db-client-executionloop-oracle-hzwvnn.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 25s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 30s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 35s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 40s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 46s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 51s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 56s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 1/1 Running 0 61s check pod test-db-client-executionloop-oracle-hzwvnn status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-executionloop-oracle-hzwvnn 0/1 Completed 0 66s check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done [ 21s ] executions total: 2710 successful: 2697 failed: 13 disconnect: 9 [ 22s ] executions total: 2711 successful: 2697 failed: 14 disconnect: 9 [2025-09-01 03:37:24.484] Connection error occurred! [2025-09-01 03:37:27.593] Connection successfully recovered! The connection was restored in 3109 milliseconds. [ 23s ] executions total: 2712 successful: 2698 failed: 14 disconnect: 9 [ 24s ] executions total: 2971 successful: 2957 failed: 14 disconnect: 9 [2025-09-01 03:37:28.747] Connection error occurred! [ 25s ] executions total: 3013 successful: 2997 failed: 16 disconnect: 10 [2025-09-01 03:37:28.747] Connection error occurred! [2025-09-01 03:37:30.821] Connection successfully recovered! The connection was restored in 2074 milliseconds. [ 26s ] executions total: 3014 successful: 2998 failed: 16 disconnect: 10 [ 27s ] executions total: 3270 successful: 3254 failed: 16 disconnect: 10 [2025-09-01 03:37:31.986] Connection error occurred! [2025-09-01 03:37:31.986] Connection error occurred! [2025-09-01 03:37:33.039] Connection successfully recovered! The connection was restored in 1053 milliseconds. [ 28s ] executions total: 3315 successful: 3298 failed: 17 disconnect: 11 [ 29s ] executions total: 3562 successful: 3545 failed: 17 disconnect: 11 [2025-09-01 03:37:34.249] Connection error occurred! [ 30s ] executions total: 3616 successful: 3597 failed: 19 disconnect: 12 [2025-09-01 03:37:34.249] Connection error occurred! [2025-09-01 03:37:36.335] Connection successfully recovered! The connection was restored in 2086 milliseconds. [ 31s ] executions total: 3617 successful: 3598 failed: 19 disconnect: 12 [ 32s ] executions total: 3869 successful: 3850 failed: 19 disconnect: 12 [2025-09-01 03:37:37.523] Connection error occurred! [2025-09-01 03:37:37.523] Connection error occurred! [2025-09-01 03:37:38.562] Connection successfully recovered! The connection was restored in 1039 milliseconds. [ 33s ] executions total: 3918 successful: 3898 failed: 20 disconnect: 13 [ 34s ] executions total: 4175 successful: 4155 failed: 20 disconnect: 13 [2025-09-01 03:37:39.721] Connection error occurred! [2025-09-01 03:37:39.721] Connection error occurred! [2025-09-01 03:37:40.785] Connection successfully recovered! The connection was restored in 1064 milliseconds. [ 35s ] executions total: 4219 successful: 4198 failed: 21 disconnect: 14 [ 36s ] executions total: 4470 successful: 4449 failed: 21 disconnect: 14 [2025-09-01 03:37:41.959] Connection error occurred! [ 37s ] executions total: 4518 successful: 4495 failed: 23 disconnect: 15 [ 38s ] executions total: 4519 successful: 4495 failed: 24 disconnect: 15 [2025-09-01 03:37:41.959] Connection error occurred! [2025-09-01 03:37:45.080] Connection successfully recovered! The connection was restored in 3121 milliseconds. [ 39s ] executions total: 4520 successful: 4496 failed: 24 disconnect: 15 [ 40s ] executions total: 4702 successful: 4678 failed: 24 disconnect: 15 [2025-09-01 03:37:46.774] Connection error occurred! [ 41s ] executions total: 4821 successful: 4795 failed: 26 disconnect: 16 [2025-09-01 03:37:46.774] Connection error occurred! [2025-09-01 03:37:48.847] Connection successfully recovered! The connection was restored in 2073 milliseconds. [ 42s ] executions total: 4822 successful: 4796 failed: 26 disconnect: 16 [ 43s ] executions total: 5085 successful: 5059 failed: 26 disconnect: 16 [2025-09-01 03:37:49.984] Connection error occurred! [ 44s ] executions total: 5123 successful: 5095 failed: 28 disconnect: 17 [ 45s ] executions total: 5124 successful: 5095 failed: 29 disconnect: 17 [ 46s ] executions total: 5125 successful: 5095 failed: 30 disconnect: 17 [2025-09-01 03:37:49.984] Connection error occurred! [2025-09-01 03:37:54.124] Connection successfully recovered! The connection was restored in 4140 milliseconds. [ 47s ] executions total: 5126 successful: 5096 failed: 30 disconnect: 17 [ 48s ] executions total: 5364 successful: 5334 failed: 30 disconnect: 17 [2025-09-01 03:37:55.364] Connection error occurred! [2025-09-01 03:37:55.364] Connection error occurred! [2025-09-01 03:37:56.403] Connection successfully recovered! The connection was restored in 1039 milliseconds. [ 49s ] executions total: 5427 successful: 5396 failed: 31 disconnect: 18 [ 50s ] executions total: 5675 successful: 5644 failed: 31 disconnect: 18 [2025-09-01 03:37:57.643] Connection error occurred! [ 51s ] executions total: 5728 successful: 5695 failed: 33 disconnect: 19 [2025-09-01 03:37:57.643] Connection error occurred! [2025-09-01 03:37:59.762] Connection successfully recovered! The connection was restored in 2119 milliseconds. [ 52s ] executions total: 5729 successful: 5696 failed: 33 disconnect: 19 [ 53s ] executions total: 5847 successful: 5814 failed: 33 disconnect: 19 [2025-09-01 03:38:01.450] Connection error occurred! [2025-09-01 03:38:01.450] Connection error occurred! [2025-09-01 03:38:02.516] Connection successfully recovered! The connection was restored in 1066 milliseconds. [ 60s ] executions total: 6028 successful: 5994 failed: 34 disconnect: 20 Test Result: Total Executions: 6028 Successful Executions: 5994 Failed Executions: 34 Disconnection Counts: 20 Connection Information: Database Type: oracle Host: oracle-hzwvnn-oracle.ns-lrqxb.svc.cluster.local Port: 1521 Database: Table: User: sys Org: Access Mode: mysql Test Type: executionloop Query: Duration: 60 seconds Interval: 1 seconds DB_CLIENT_BATCH_DATA_COUNT: 5994 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-executionloop-oracle-hzwvnn --namespace ns-lrqxb ` pod/test-db-client-executionloop-oracle-hzwvnn patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-executionloop-oracle-hzwvnn" force deleted test failover drainnode check cluster status before cluster-failover-drainnode check cluster status done cluster_status:Running check node drain check node drain success kubectl get pod oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-15164480-vmss000000 success check if multiple pods are on the same node kubectl get pod oracle-hzwvnn-oracle-1 --namespace ns-lrqxb -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-15164480-vmss000000 success Multiple pods on the same node test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn created apply test-chaos-mesh-podkill-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podkill-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:38 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn created apply test-chaos-mesh-podkill-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podkill-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:39 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn patched failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn created apply test-chaos-mesh-podkill-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podkill-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:40 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn patched failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn created apply test-chaos-mesh-podkill-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podkill-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:41 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn patched failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn created apply test-chaos-mesh-podkill-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podkill-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:42 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn patched failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn created apply test-chaos-mesh-podkill-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podkill-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:36 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:43 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:19 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podkill-oracle-hzwvnn patched check failover pod name timeout `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop oracle-hzwvnn --auto-approve --force=true --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-stop-q6f66 created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-stop-q6f66 -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-stop-q6f66 ns-lrqxb Stop oracle-hzwvnn Creating -/- Sep 01,2025 11:44 UTC+0800 ops_status:oracle-hzwvnn-stop-q6f66 ns-lrqxb Stop oracle-hzwvnn Creating -/- Sep 01,2025 11:44 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Stopping Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping cluster_status:Stopping check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-stop-q6f66 ns-lrqxb Stop oracle-hzwvnn observer,oracle Succeed 3/3 Sep 01,2025 11:44 UTC+0800 check ops status done ops_status:oracle-hzwvnn-stop-q6f66 ns-lrqxb Stop oracle-hzwvnn observer,oracle Succeed 3/3 Sep 01,2025 11:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-stop-q6f66 --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-stop-q6f66 patched `kbcli cluster delete-ops --name oracle-hzwvnn-stop-q6f66 --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-stop-q6f66 deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start oracle-hzwvnn --force=true --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-start-j2f55 created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-start-j2f55 -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-start-j2f55 ns-lrqxb Start oracle-hzwvnn Sep 01,2025 11:44 UTC+0800 ops_status:oracle-hzwvnn-start-j2f55 ns-lrqxb Start oracle-hzwvnn Creating -/- Sep 01,2025 11:44 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-start-j2f55 ns-lrqxb Start oracle-hzwvnn observer,oracle Succeed 3/3 Sep 01,2025 11:44 UTC+0800 check ops status done ops_status:oracle-hzwvnn-start-j2f55 ns-lrqxb Start oracle-hzwvnn observer,oracle Succeed 3/3 Sep 01,2025 11:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-start-j2f55 --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-start-j2f55 patched `kbcli cluster delete-ops --name oracle-hzwvnn-start-j2f55 --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-start-j2f55 deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test switchover cluster promote check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster promote oracle-hzwvnn --auto-approve --force=true --component oracle --instance oracle-hzwvnn-oracle-1 --namespace ns-lrqxb ` component:oracle OpsRequest oracle-hzwvnn-switchover-4w4pc created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-switchover-4w4pc -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Creating -/- Sep 01,2025 11:53 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 ops_status:oracle-hzwvnn-switchover-4w4pc ns-lrqxb Switchover oracle-hzwvnn oracle Running 0/1 Sep 01,2025 11:53 UTC+0800 check ops status timeout --------------------------------------get cluster oracle-hzwvnn yaml-------------------------------------- `kubectl get cluster oracle-hzwvnn -o yaml --namespace ns-lrqxb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"oracle-hzwvnn-switchover-4w4pc","type":"Switchover"***]' kubeblocks.io/reconcile: "2025-09-01T03:53:25.898332134Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"oracle-hzwvnn","namespace":"ns-lrqxb"***,"spec":***"clusterDefinitionRef":"oracle","componentSpecs":[***"disableExporter":false,"monitor":true,"name":"oracle","replicas":2,"resources":***"limits":***"cpu":"4000m","memory":"8Gi"***,"requests":***"cpu":"4000m","memory":"8Gi"***,"serviceVersion":"12.2.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,***"name":"fra","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***]***,***"name":"observer","replicas":1,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"12.2.0"***],"terminationPolicy":"Delete","topology":"replication"*** creationTimestamp: "2025-09-01T03:19:20Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: oracle-hzwvnn clusterdefinition.kubeblocks.io/name: oracle clusterversion.kubeblocks.io/name: "" name: oracle-hzwvnn namespace: ns-lrqxb resourceVersion: "39287" uid: 302b65fe-03dc-4b9a-8045-6498e180e60d spec: clusterDefinitionRef: oracle componentSpecs: - componentDef: oracle-12c disableExporter: false monitor: true name: oracle replicas: 2 resources: limits: cpu: "4" memory: 8Gi requests: cpu: "4" memory: 8Gi serviceVersion: 12.2.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: fra spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - componentDef: oracle-observer-12c name: observer replicas: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi serviceVersion: 12.2.0 resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Delete topology: replication status: clusterDefGeneration: 1 components: observer: message: InstanceSet/oracle-hzwvnn-observer: '["oracle-hzwvnn-observer-0"]' phase: Running podsReady: true podsReadyTime: "2025-09-01T03:53:49Z" oracle: phase: Running podsReady: true podsReadyTime: "2025-09-01T03:52:32Z" conditions: - lastTransitionTime: "2025-09-01T03:19:20Z" message: 'The operator has started the provisioning of Cluster: oracle-hzwvnn' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:19:20Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T03:53:49Z" message: all pods of components are ready, waiting for the probe detection successful reason: AllReplicasReady status: "True" type: ReplicasReady - lastTransitionTime: "2025-09-01T03:53:49Z" message: 'Cluster: oracle-hzwvnn is ready, current phase is Running' reason: ClusterReady status: "True" type: Ready observedGeneration: 5 phase: Running ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster oracle-hzwvnn-------------------------------------- `kubectl describe cluster oracle-hzwvnn --namespace ns-lrqxb ` Name: oracle-hzwvnn Namespace: ns-lrqxb Labels: app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"oracle-hzwvnn-switchover-4w4pc","type":"Switchover"***] kubeblocks.io/reconcile: 2025-09-01T03:53:25.898332134Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:19:20Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 39287 UID: 302b65fe-03dc-4b9a-8045-6498e180e60d Spec: Cluster Definition Ref: oracle Component Specs: Component Def: oracle-12c Disable Exporter: false Monitor: true Name: oracle Replicas: 2 Resources: Limits: Cpu: 4 Memory: 8Gi Requests: Cpu: 4 Memory: 8Gi Service Version: 12.2.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: fra Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def: oracle-observer-12c Name: observer Replicas: 1 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Service Version: 12.2.0 Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Topology: replication Status: Cluster Def Generation: 1 Components: Observer: Message: InstanceSet/oracle-hzwvnn-observer: ["oracle-hzwvnn-observer-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:53:49Z Oracle: Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:52:32Z Conditions: Last Transition Time: 2025-09-01T03:19:20Z Message: The operator has started the provisioning of Cluster: oracle-hzwvnn Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:19:20Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T03:53:49Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2025-09-01T03:53:49Z Message: Cluster: oracle-hzwvnn is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 5 Phase: Running Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 41m (x2 over 41m) cluster-controller The operator has started the provisioning of Cluster: oracle-hzwvnn Normal ApplyResourcesSucceed 41m (x2 over 41m) cluster-controller Successfully applied for resources Warning Unhealthy 34m event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning Unhealthy 28m (x5 over 33m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: MOUNTED Warning Unhealthy 27m (x4 over 31m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01012: not logged on Process ID: 0 Session ID: 0 Serial number: 0 Warning Unhealthy 27m (x16 over 33m) event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning ReplicasNotReady 24m cluster-controller pods are not ready in Components: [observer], refer to related component message in Cluster.status.components Warning ComponentsNotReady 24m cluster-controller pods are unavailable in Components: [observer], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 24m (x2 over 41m) cluster-controller component is Creating Warning ComponentsNotReady 23m cluster-controller pods are unavailable in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ReplicasNotReady 23m cluster-controller pods are not ready in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ComponentsNotReady 22m (x2 over 23m) cluster-controller pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Normal Running 22m (x2 over 24m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal ClusterReady 22m (x2 over 24m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal AllReplicasReady 22m (x2 over 24m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ComponentPhaseTransition 22m (x3 over 23m) cluster-controller component is Updating Warning ReplicasNotReady 22m (x3 over 23m) cluster-controller pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 21m (x5 over 24m) cluster-controller component is Running Normal ComponentPhaseTransition 16m cluster-controller component is Stopped Normal HorizontalScale 16m component-controller start horizontal scale component observer of cluster oracle-hzwvnn from 0 to 1 Normal HorizontalScale 16m component-controller start horizontal scale component oracle of cluster oracle-hzwvnn from 0 to 2 Warning FailedAttachVolume 13m event-controller Pod oracle-hzwvnn-oracle-1: AttachVolume.Attach failed for volume "pvc-ad2ac486-f903-4c1c-b7f9-ef3a94e2fa9a" : volume attachment is being deleted Normal ComponentPhaseTransition 11m (x4 over 15m) cluster-controller component is Failed Warning BackOff 9m1s (x19 over 14m) event-controller Pod oracle-hzwvnn-observer-0: Back-off restarting failed container oracle-observer in pod oracle-hzwvnn-observer-0_ns-lrqxb(d9984136-8d01-4a86-850e-8d0cb92d82af) ------------------------------------------------------------------------------------------------------------------ `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-switchover-4w4pc --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-switchover-4w4pc patched `kbcli cluster delete-ops --name oracle-hzwvnn-switchover-4w4pc --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-switchover-4w4pc deleted switchover pod:oracle-hzwvnn-oracle-0 checking switchover... check switchover timeout test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn created apply test-chaos-mesh-oom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-oom-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster oracle-hzwvnn yaml-------------------------------------- `kubectl get cluster oracle-hzwvnn -o yaml --namespace ns-lrqxb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-09-01T04:08:52.950810885Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"oracle-hzwvnn","namespace":"ns-lrqxb"***,"spec":***"clusterDefinitionRef":"oracle","componentSpecs":[***"disableExporter":false,"monitor":true,"name":"oracle","replicas":2,"resources":***"limits":***"cpu":"4000m","memory":"8Gi"***,"requests":***"cpu":"4000m","memory":"8Gi"***,"serviceVersion":"12.2.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,***"name":"fra","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***]***,***"name":"observer","replicas":1,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"12.2.0"***],"terminationPolicy":"Delete","topology":"replication"*** creationTimestamp: "2025-09-01T03:19:20Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: oracle-hzwvnn clusterdefinition.kubeblocks.io/name: oracle clusterversion.kubeblocks.io/name: "" name: oracle-hzwvnn namespace: ns-lrqxb resourceVersion: "48667" uid: 302b65fe-03dc-4b9a-8045-6498e180e60d spec: clusterDefinitionRef: oracle componentSpecs: - componentDef: oracle-12c disableExporter: false monitor: true name: oracle replicas: 2 resources: limits: cpu: "4" memory: 8Gi requests: cpu: "4" memory: 8Gi serviceVersion: 12.2.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: fra spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - componentDef: oracle-observer-12c name: observer replicas: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi serviceVersion: 12.2.0 resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Delete topology: replication status: clusterDefGeneration: 1 components: observer: message: InstanceSet/oracle-hzwvnn-observer: '["oracle-hzwvnn-observer-0"]' phase: Running podsReady: true podsReadyTime: "2025-09-01T03:53:49Z" oracle: phase: Updating podsReady: false podsReadyTime: "2025-09-01T03:52:32Z" conditions: - lastTransitionTime: "2025-09-01T03:19:20Z" message: 'The operator has started the provisioning of Cluster: oracle-hzwvnn' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:19:20Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T04:01:24Z" message: 'pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T04:01:24Z" message: 'pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 5 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster oracle-hzwvnn-------------------------------------- `kubectl describe cluster oracle-hzwvnn --namespace ns-lrqxb ` Name: oracle-hzwvnn Namespace: ns-lrqxb Labels: app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-09-01T04:08:52.950810885Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:19:20Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 48667 UID: 302b65fe-03dc-4b9a-8045-6498e180e60d Spec: Cluster Definition Ref: oracle Component Specs: Component Def: oracle-12c Disable Exporter: false Monitor: true Name: oracle Replicas: 2 Resources: Limits: Cpu: 4 Memory: 8Gi Requests: Cpu: 4 Memory: 8Gi Service Version: 12.2.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: fra Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def: oracle-observer-12c Name: observer Replicas: 1 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Service Version: 12.2.0 Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Topology: replication Status: Cluster Def Generation: 1 Components: Observer: Message: InstanceSet/oracle-hzwvnn-observer: ["oracle-hzwvnn-observer-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:53:49Z Oracle: Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T03:52:32Z Conditions: Last Transition Time: 2025-09-01T03:19:20Z Message: The operator has started the provisioning of Cluster: oracle-hzwvnn Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:19:20Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T04:01:24Z Message: pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T04:01:24Z Message: pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 5 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 49m (x2 over 49m) cluster-controller The operator has started the provisioning of Cluster: oracle-hzwvnn Normal ApplyResourcesSucceed 49m (x2 over 49m) cluster-controller Successfully applied for resources Warning Unhealthy 41m event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning Unhealthy 36m (x5 over 41m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: MOUNTED Warning Unhealthy 34m (x4 over 39m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01012: not logged on Process ID: 0 Session ID: 0 Serial number: 0 Warning Unhealthy 34m (x16 over 40m) event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning ReplicasNotReady 32m cluster-controller pods are not ready in Components: [observer], refer to related component message in Cluster.status.components Warning ComponentsNotReady 32m cluster-controller pods are unavailable in Components: [observer], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 32m (x2 over 49m) cluster-controller component is Creating Warning ComponentsNotReady 30m cluster-controller pods are unavailable in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ReplicasNotReady 30m cluster-controller pods are not ready in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ComponentsNotReady 30m (x2 over 30m) cluster-controller pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Normal Running 29m (x2 over 32m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal ClusterReady 29m (x2 over 32m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal AllReplicasReady 29m (x2 over 32m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ComponentPhaseTransition 28m (x5 over 32m) cluster-controller component is Running Normal ComponentPhaseTransition 24m cluster-controller component is Stopped Normal HorizontalScale 24m component-controller start horizontal scale component observer of cluster oracle-hzwvnn from 0 to 1 Normal HorizontalScale 23m component-controller start horizontal scale component oracle of cluster oracle-hzwvnn from 0 to 2 Warning FailedAttachVolume 21m event-controller Pod oracle-hzwvnn-oracle-1: AttachVolume.Attach failed for volume "pvc-ad2ac486-f903-4c1c-b7f9-ef3a94e2fa9a" : volume attachment is being deleted Normal ComponentPhaseTransition 19m (x4 over 22m) cluster-controller component is Failed Warning BackOff 16m (x19 over 22m) event-controller Pod oracle-hzwvnn-observer-0: Back-off restarting failed container oracle-observer in pod oracle-hzwvnn-observer-0_ns-lrqxb(d9984136-8d01-4a86-850e-8d0cb92d82af) Normal ComponentPhaseTransition 7m30s (x21 over 30m) cluster-controller component is Updating Warning ReplicasNotReady 7m30s (x14 over 30m) cluster-controller pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Warning Unhealthy 104s (x9 over 4m26s) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01034: ORACLE not available Process ID: 0 Session ID: 0 Serial number: 0 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster status again `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found check failover pod name get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get pod name timeout failover pod name: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn created apply test-chaos-mesh-oom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-oom-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn patched failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn created apply test-chaos-mesh-oom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-oom-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster oracle-hzwvnn yaml-------------------------------------- `kubectl get cluster oracle-hzwvnn -o yaml --namespace ns-lrqxb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-09-01T04:14:12.355172359Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"oracle-hzwvnn","namespace":"ns-lrqxb"***,"spec":***"clusterDefinitionRef":"oracle","componentSpecs":[***"disableExporter":false,"monitor":true,"name":"oracle","replicas":2,"resources":***"limits":***"cpu":"4000m","memory":"8Gi"***,"requests":***"cpu":"4000m","memory":"8Gi"***,"serviceVersion":"12.2.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,***"name":"fra","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***]***,***"name":"observer","replicas":1,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"12.2.0"***],"terminationPolicy":"Delete","topology":"replication"*** creationTimestamp: "2025-09-01T03:19:20Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: oracle-hzwvnn clusterdefinition.kubeblocks.io/name: oracle clusterversion.kubeblocks.io/name: "" name: oracle-hzwvnn namespace: ns-lrqxb resourceVersion: "51047" uid: 302b65fe-03dc-4b9a-8045-6498e180e60d spec: clusterDefinitionRef: oracle componentSpecs: - componentDef: oracle-12c disableExporter: false monitor: true name: oracle replicas: 2 resources: limits: cpu: "4" memory: 8Gi requests: cpu: "4" memory: 8Gi serviceVersion: 12.2.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: fra spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - componentDef: oracle-observer-12c name: observer replicas: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi serviceVersion: 12.2.0 resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Delete topology: replication status: clusterDefGeneration: 1 components: observer: message: InstanceSet/oracle-hzwvnn-observer: '["oracle-hzwvnn-observer-0"]' phase: Running podsReady: true podsReadyTime: "2025-09-01T03:53:49Z" oracle: message: InstanceSet/oracle-hzwvnn-oracle: '["oracle-hzwvnn-oracle-0"]' phase: Updating podsReady: false podsReadyTime: "2025-09-01T04:10:31Z" conditions: - lastTransitionTime: "2025-09-01T03:19:20Z" message: 'The operator has started the provisioning of Cluster: oracle-hzwvnn' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:19:20Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T04:10:40Z" message: 'pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T04:10:40Z" message: 'pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 5 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster oracle-hzwvnn-------------------------------------- `kubectl describe cluster oracle-hzwvnn --namespace ns-lrqxb ` Name: oracle-hzwvnn Namespace: ns-lrqxb Labels: app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-09-01T04:14:12.355172359Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:19:20Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 51047 UID: 302b65fe-03dc-4b9a-8045-6498e180e60d Spec: Cluster Definition Ref: oracle Component Specs: Component Def: oracle-12c Disable Exporter: false Monitor: true Name: oracle Replicas: 2 Resources: Limits: Cpu: 4 Memory: 8Gi Requests: Cpu: 4 Memory: 8Gi Service Version: 12.2.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: fra Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def: oracle-observer-12c Name: observer Replicas: 1 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Service Version: 12.2.0 Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Topology: replication Status: Cluster Def Generation: 1 Components: Observer: Message: InstanceSet/oracle-hzwvnn-observer: ["oracle-hzwvnn-observer-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:53:49Z Oracle: Message: InstanceSet/oracle-hzwvnn-oracle: ["oracle-hzwvnn-oracle-0"] Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T04:10:31Z Conditions: Last Transition Time: 2025-09-01T03:19:20Z Message: The operator has started the provisioning of Cluster: oracle-hzwvnn Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:19:20Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T04:10:40Z Message: pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T04:10:40Z Message: pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 5 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal PreCheckSucceed 58m (x2 over 58m) cluster-controller The operator has started the provisioning of Cluster: oracle-hzwvnn Normal ApplyResourcesSucceed 58m (x2 over 58m) cluster-controller Successfully applied for resources Warning Unhealthy 51m event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning Unhealthy 45m (x5 over 50m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: MOUNTED Warning Unhealthy 44m (x4 over 48m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01012: not logged on Process ID: 0 Session ID: 0 Serial number: 0 Warning Unhealthy 43m (x16 over 49m) event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning ReplicasNotReady 41m cluster-controller pods are not ready in Components: [observer], refer to related component message in Cluster.status.components Warning ComponentsNotReady 41m cluster-controller pods are unavailable in Components: [observer], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 41m (x2 over 58m) cluster-controller component is Creating Warning ComponentsNotReady 40m cluster-controller pods are unavailable in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ReplicasNotReady 40m cluster-controller pods are not ready in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ComponentsNotReady 39m (x2 over 40m) cluster-controller pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Normal Running 39m (x2 over 41m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal ClusterReady 39m (x2 over 41m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal AllReplicasReady 39m (x2 over 41m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ComponentPhaseTransition 38m (x5 over 41m) cluster-controller component is Running Normal ComponentPhaseTransition 33m cluster-controller component is Stopped Normal HorizontalScale 33m component-controller start horizontal scale component observer of cluster oracle-hzwvnn from 0 to 1 Normal HorizontalScale 33m component-controller start horizontal scale component oracle of cluster oracle-hzwvnn from 0 to 2 Warning FailedAttachVolume 30m event-controller Pod oracle-hzwvnn-oracle-1: AttachVolume.Attach failed for volume "pvc-ad2ac486-f903-4c1c-b7f9-ef3a94e2fa9a" : volume attachment is being deleted Warning BackOff 25m (x19 over 31m) event-controller Pod oracle-hzwvnn-observer-0: Back-off restarting failed container oracle-observer in pod oracle-hzwvnn-observer-0_ns-lrqxb(d9984136-8d01-4a86-850e-8d0cb92d82af) Warning ReplicasNotReady 16m (x14 over 40m) cluster-controller pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 16m (x21 over 40m) cluster-controller component is Updating Normal ComponentPhaseTransition 8m33s (x6 over 32m) cluster-controller component is Failed Warning Abnormal 8m33s (x6 over 32m) cluster-controller Cluster: oracle-hzwvnn is Abnormal, check according to the components message Warning Unhealthy 5m29s (x16 over 13m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01034: ORACLE not available Process ID: 0 Session ID: 0 Serial number: 0 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster status again `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get pod name timeout failover pod name: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn created apply test-chaos-mesh-oom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-oom-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn patched failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn created apply test-chaos-mesh-oom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-oom-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster oracle-hzwvnn yaml-------------------------------------- `kubectl get cluster oracle-hzwvnn -o yaml --namespace ns-lrqxb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-09-01T04:23:57.911692691Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"oracle-hzwvnn","namespace":"ns-lrqxb"***,"spec":***"clusterDefinitionRef":"oracle","componentSpecs":[***"disableExporter":false,"monitor":true,"name":"oracle","replicas":2,"resources":***"limits":***"cpu":"4000m","memory":"8Gi"***,"requests":***"cpu":"4000m","memory":"8Gi"***,"serviceVersion":"12.2.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,***"name":"fra","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***]***,***"name":"observer","replicas":1,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"12.2.0"***],"terminationPolicy":"Delete","topology":"replication"*** creationTimestamp: "2025-09-01T03:19:20Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 5 labels: app.kubernetes.io/instance: oracle-hzwvnn clusterdefinition.kubeblocks.io/name: oracle clusterversion.kubeblocks.io/name: "" name: oracle-hzwvnn namespace: ns-lrqxb resourceVersion: "55300" uid: 302b65fe-03dc-4b9a-8045-6498e180e60d spec: clusterDefinitionRef: oracle componentSpecs: - componentDef: oracle-12c disableExporter: false monitor: true name: oracle replicas: 2 resources: limits: cpu: "4" memory: 8Gi requests: cpu: "4" memory: 8Gi serviceVersion: 12.2.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: fra spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - componentDef: oracle-observer-12c name: observer replicas: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi serviceVersion: 12.2.0 resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Delete topology: replication status: clusterDefGeneration: 1 components: observer: message: InstanceSet/oracle-hzwvnn-observer: '["oracle-hzwvnn-observer-0"]' phase: Running podsReady: true podsReadyTime: "2025-09-01T03:53:49Z" oracle: message: InstanceSet/oracle-hzwvnn-oracle: '["oracle-hzwvnn-oracle-0"]' phase: Updating podsReady: false podsReadyTime: "2025-09-01T04:20:18Z" conditions: - lastTransitionTime: "2025-09-01T03:19:20Z" message: 'The operator has started the provisioning of Cluster: oracle-hzwvnn' observedGeneration: 5 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:19:20Z" message: Successfully applied for resources observedGeneration: 5 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T04:20:26Z" message: 'pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T04:20:26Z" message: 'pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 5 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster oracle-hzwvnn-------------------------------------- `kubectl describe cluster oracle-hzwvnn --namespace ns-lrqxb ` Name: oracle-hzwvnn Namespace: ns-lrqxb Labels: app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-09-01T04:23:57.911692691Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:19:20Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 5 Resource Version: 55300 UID: 302b65fe-03dc-4b9a-8045-6498e180e60d Spec: Cluster Definition Ref: oracle Component Specs: Component Def: oracle-12c Disable Exporter: false Monitor: true Name: oracle Replicas: 2 Resources: Limits: Cpu: 4 Memory: 8Gi Requests: Cpu: 4 Memory: 8Gi Service Version: 12.2.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: fra Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Component Def: oracle-observer-12c Name: observer Replicas: 1 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Service Version: 12.2.0 Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Topology: replication Status: Cluster Def Generation: 1 Components: Observer: Message: InstanceSet/oracle-hzwvnn-observer: ["oracle-hzwvnn-observer-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T03:53:49Z Oracle: Message: InstanceSet/oracle-hzwvnn-oracle: ["oracle-hzwvnn-oracle-0"] Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T04:20:18Z Conditions: Last Transition Time: 2025-09-01T03:19:20Z Message: The operator has started the provisioning of Cluster: oracle-hzwvnn Observed Generation: 5 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:19:20Z Message: Successfully applied for resources Observed Generation: 5 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T04:20:26Z Message: pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T04:20:26Z Message: pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 5 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning Unhealthy 55m (x5 over 60m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: MOUNTED Warning Unhealthy 53m (x4 over 58m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01012: not logged on Process ID: 0 Session ID: 0 Serial number: 0 Warning Unhealthy 53m (x16 over 59m) event-controller Pod oracle-hzwvnn-oracle-1: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01507: database not mounted Warning ComponentsNotReady 51m cluster-controller pods are unavailable in Components: [observer], refer to related component message in Cluster.status.components Warning ReplicasNotReady 51m cluster-controller pods are not ready in Components: [observer], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 51m (x2 over 68m) cluster-controller component is Creating Warning ReplicasNotReady 49m cluster-controller pods are not ready in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ComponentsNotReady 49m cluster-controller pods are unavailable in Components: [observer oracle], refer to related component message in Cluster.status.components Warning ComponentsNotReady 49m (x2 over 49m) cluster-controller pods are unavailable in Components: [oracle], refer to related component message in Cluster.status.components Normal AllReplicasReady 48m (x2 over 51m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 48m (x2 over 51m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal Running 48m (x2 over 51m) cluster-controller Cluster: oracle-hzwvnn is ready, current phase is Running Normal ComponentPhaseTransition 47m (x5 over 51m) cluster-controller component is Running Normal ComponentPhaseTransition 43m cluster-controller component is Stopped Normal HorizontalScale 43m component-controller start horizontal scale component observer of cluster oracle-hzwvnn from 0 to 1 Normal HorizontalScale 42m component-controller start horizontal scale component oracle of cluster oracle-hzwvnn from 0 to 2 Warning FailedAttachVolume 40m event-controller Pod oracle-hzwvnn-oracle-1: AttachVolume.Attach failed for volume "pvc-ad2ac486-f903-4c1c-b7f9-ef3a94e2fa9a" : volume attachment is being deleted Warning BackOff 35m (x19 over 41m) event-controller Pod oracle-hzwvnn-observer-0: Back-off restarting failed container oracle-observer in pod oracle-hzwvnn-observer-0_ns-lrqxb(d9984136-8d01-4a86-850e-8d0cb92d82af) Warning ReplicasNotReady 26m (x14 over 49m) cluster-controller pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Warning Abnormal 18m (x6 over 41m) cluster-controller Cluster: oracle-hzwvnn is Abnormal, check according to the components message Normal ComponentPhaseTransition 9m4s (x7 over 41m) cluster-controller component is Failed Normal ComponentPhaseTransition 8m21s (x24 over 49m) cluster-controller component is Updating Warning Unhealthy 4m (x22 over 23m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01034: ORACLE not available Process ID: 0 Session ID: 0 Serial number: 0 ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster status again `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn patched get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get pod name timeout failover pod name: `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn created apply test-chaos-mesh-oom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-oom-oracle-hzwvnn.yaml` check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "oracle-hzwvnn-oracle-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "oracle-hzwvnn-oracle-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:33 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-oracle-hzwvnn" force deleted stresschaos.chaos-mesh.org/test-chaos-mesh-oom-oracle-hzwvnn patched check failover pod name timeout test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-oracle-hzwvnn" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-oracle-hzwvnn.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-oracle-hzwvnn created apply test-chaos-mesh-dnserror-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-dnserror-oracle-hzwvnn.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:44 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:33 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 11:45 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-oracle-hzwvnn" force deleted dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover dnserror Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart oracle-hzwvnn --auto-approve --force=true --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-restart-grkc9 created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-restart-grkc9 -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-restart-grkc9 ns-lrqxb Restart oracle-hzwvnn oracle,observer Creating -/- Sep 01,2025 12:36 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-restart-grkc9 ns-lrqxb Restart oracle-hzwvnn oracle,observer Succeed 3/3 Sep 01,2025 12:36 UTC+0800 check ops status done ops_status:oracle-hzwvnn-restart-grkc9 ns-lrqxb Restart oracle-hzwvnn oracle,observer Succeed 3/3 Sep 01,2025 12:36 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-restart-grkc9 --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-restart-grkc9 patched `kbcli cluster delete-ops --name oracle-hzwvnn-restart-grkc9 --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-restart-grkc9 deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover networkbandwidth check cluster status before cluster-failover-networkbandwidth check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidth-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidth-oracle-hzwvnn" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidth-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidth-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidth-oracle-hzwvnn.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidth-oracle-hzwvnn created apply test-chaos-mesh-networkbandwidth-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-networkbandwidth-oracle-hzwvnn.yaml` networkbandwidth chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidth-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidth-oracle-hzwvnn" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidth-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover networkbandwidth Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover networkcorrupt check cluster status before cluster-failover-networkcorrupt check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorrupt-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorrupt-oracle-hzwvnn" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorrupt-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorrupt-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorrupt-oracle-hzwvnn.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorrupt-oracle-hzwvnn created apply test-chaos-mesh-networkcorrupt-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-networkcorrupt-oracle-hzwvnn.yaml` networkcorrupt chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorrupt-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorrupt-oracle-hzwvnn" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorrupt-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover networkcorrupt Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success check component observer exists `kubectl get components -l app.kubernetes.io/instance=oracle-hzwvnn,apps.kubeblocks.io/component-name=observer --namespace ns-lrqxb | (grep "observer" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in oracle-hzwvnn namespace. `kbcli cluster hscale oracle-hzwvnn --auto-approve --force=true --components observer --replicas 2 --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-horizontalscaling-69p6c created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-horizontalscaling-69p6c -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-horizontalscaling-69p6c ns-lrqxb HorizontalScaling oracle-hzwvnn observer Creating -/- Sep 01,2025 12:43 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-observer-1 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:43 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done No resources found in oracle-hzwvnn namespace. check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-horizontalscaling-69p6c ns-lrqxb HorizontalScaling oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 12:43 UTC+0800 check ops status done ops_status:oracle-hzwvnn-horizontalscaling-69p6c ns-lrqxb HorizontalScaling oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 12:43 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-horizontalscaling-69p6c --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-horizontalscaling-69p6c patched `kbcli cluster delete-ops --name oracle-hzwvnn-horizontalscaling-69p6c --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-horizontalscaling-69p6c deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success check component observer exists `kubectl get components -l app.kubernetes.io/instance=oracle-hzwvnn,apps.kubeblocks.io/component-name=observer --namespace ns-lrqxb | (grep "observer" || true )` cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in oracle-hzwvnn namespace. `kbcli cluster hscale oracle-hzwvnn --auto-approve --force=true --components observer --replicas 1 --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-horizontalscaling-ph8kc created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-horizontalscaling-ph8kc -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-horizontalscaling-ph8kc ns-lrqxb HorizontalScaling oracle-hzwvnn observer Pending -/- Sep 01,2025 12:44 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done No resources found in oracle-hzwvnn namespace. check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-horizontalscaling-ph8kc ns-lrqxb HorizontalScaling oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 12:44 UTC+0800 check ops status done ops_status:oracle-hzwvnn-horizontalscaling-ph8kc ns-lrqxb HorizontalScaling oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 12:44 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-horizontalscaling-ph8kc --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-horizontalscaling-ph8kc patched `kbcli cluster delete-ops --name oracle-hzwvnn-horizontalscaling-ph8kc --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-horizontalscaling-ph8kc deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover networkloss check cluster status before cluster-failover-networkloss check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkloss-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-oracle-hzwvnn" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkloss-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkloss-oracle-hzwvnn.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkloss-oracle-hzwvnn created apply test-chaos-mesh-networkloss-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-networkloss-oracle-hzwvnn.yaml` networkloss chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:20Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:20Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkloss-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-oracle-hzwvnn" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkloss-oracle-hzwvnn" not found check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover networkloss Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=oracle-hzwvnn,apps.kubeblocks.io/component-name=oracle,apps.kubeblocks.io/vct-name=fra --namespace ns-lrqxb ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in oracle-hzwvnn namespace. `kbcli cluster volume-expand oracle-hzwvnn --auto-approve --force=true --components oracle --volume-claim-templates fra --storage 21Gi --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-volumeexpansion-qmlfv created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-volumeexpansion-qmlfv -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-volumeexpansion-qmlfv ns-lrqxb VolumeExpansion oracle-hzwvnn oracle Creating -/- Sep 01,2025 12:47 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4 / 4 8Gi / 8Gi data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:36 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done No resources found in oracle-hzwvnn namespace. check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-volumeexpansion-qmlfv ns-lrqxb VolumeExpansion oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 12:47 UTC+0800 check ops status done ops_status:oracle-hzwvnn-volumeexpansion-qmlfv ns-lrqxb VolumeExpansion oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 12:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-volumeexpansion-qmlfv --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-volumeexpansion-qmlfv patched `kbcli cluster delete-ops --name oracle-hzwvnn-volumeexpansion-qmlfv --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-volumeexpansion-qmlfv deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale oracle-hzwvnn --auto-approve --force=true --components oracle --cpu 4100m --memory 8.1Gi --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-verticalscaling-wg5cd created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-verticalscaling-wg5cd -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-verticalscaling-wg5cd ns-lrqxb VerticalScaling oracle-hzwvnn oracle Pending -/- Sep 01,2025 12:52 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:53 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-verticalscaling-wg5cd ns-lrqxb VerticalScaling oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 12:52 UTC+0800 check ops status done ops_status:oracle-hzwvnn-verticalscaling-wg5cd ns-lrqxb VerticalScaling oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 12:52 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-verticalscaling-wg5cd --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-verticalscaling-wg5cd patched `kbcli cluster delete-ops --name oracle-hzwvnn-verticalscaling-wg5cd --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-verticalscaling-wg5cd deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running check node drain check node drain success `kill 1` Defaulted container "oracle" out of: oracle, exporter, lorry, config-manager, oracle-init-container (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:53 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get oracle primary pod name... get pod name timeout check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it --namespace ns-lrqxb -- bash ` connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... `kubectl delete pod -n ns-lrqxb oracle-hzwvnn-oracle-0 --force ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "oracle-hzwvnn-oracle-0" force deleted connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... connect checking... check cluster connect timeout --------------------------------------get cluster oracle-hzwvnn yaml-------------------------------------- `kubectl get cluster oracle-hzwvnn -o yaml --namespace ns-lrqxb ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/reconcile: "2025-09-01T04:43:26.096895492Z" kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"oracle-hzwvnn","namespace":"ns-lrqxb"***,"spec":***"clusterDefinitionRef":"oracle","componentSpecs":[***"disableExporter":false,"monitor":true,"name":"oracle","replicas":2,"resources":***"limits":***"cpu":"4000m","memory":"8Gi"***,"requests":***"cpu":"4000m","memory":"8Gi"***,"serviceVersion":"12.2.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***,***"name":"fra","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"20Gi"***]***,***"name":"observer","replicas":1,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"12.2.0"***],"terminationPolicy":"Delete","topology":"replication"*** creationTimestamp: "2025-09-01T03:19:20Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 9 labels: app.kubernetes.io/instance: oracle-hzwvnn clusterdefinition.kubeblocks.io/name: oracle clusterversion.kubeblocks.io/name: "" name: oracle-hzwvnn namespace: ns-lrqxb resourceVersion: "71573" uid: 302b65fe-03dc-4b9a-8045-6498e180e60d spec: clusterDefinitionRef: oracle componentSpecs: - componentDef: oracle-12c disableExporter: false monitor: true name: oracle replicas: 2 resources: limits: cpu: 4100m memory: 8697308774400m requests: cpu: 4100m memory: 8697308774400m serviceVersion: 12.2.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 20Gi - name: fra spec: accessModes: - ReadWriteOnce resources: requests: storage: 21Gi - componentDef: oracle-observer-12c name: observer replicas: 1 resources: limits: cpu: 500m memory: 512Mi requests: cpu: 500m memory: 512Mi serviceVersion: 12.2.0 resources: cpu: "0" memory: "0" storage: size: "0" terminationPolicy: Delete topology: replication status: clusterDefGeneration: 1 components: observer: message: InstanceSet/oracle-hzwvnn-observer: '["oracle-hzwvnn-observer-0"]' phase: Running podsReady: true podsReadyTime: "2025-09-01T04:44:42Z" oracle: message: InstanceSet/oracle-hzwvnn-oracle: '["oracle-hzwvnn-oracle-0"]' phase: Running podsReady: true podsReadyTime: "2025-09-01T04:59:29Z" conditions: - lastTransitionTime: "2025-09-01T03:19:20Z" message: 'The operator has started the provisioning of Cluster: oracle-hzwvnn' observedGeneration: 9 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T04:52:06Z" message: Successfully applied for resources observedGeneration: 9 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T04:59:29Z" message: all pods of components are ready, waiting for the probe detection successful reason: AllReplicasReady status: "True" type: ReplicasReady - lastTransitionTime: "2025-09-01T04:59:29Z" message: 'Cluster: oracle-hzwvnn is ready, current phase is Running' reason: ClusterReady status: "True" type: Ready observedGeneration: 9 phase: Running ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster oracle-hzwvnn-------------------------------------- `kubectl describe cluster oracle-hzwvnn --namespace ns-lrqxb ` Name: oracle-hzwvnn Namespace: ns-lrqxb Labels: app.kubernetes.io/instance=oracle-hzwvnn clusterdefinition.kubeblocks.io/name=oracle clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/reconcile: 2025-09-01T04:43:26.096895492Z API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:19:20Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 9 Resource Version: 71573 UID: 302b65fe-03dc-4b9a-8045-6498e180e60d Spec: Cluster Definition Ref: oracle Component Specs: Component Def: oracle-12c Disable Exporter: false Monitor: true Name: oracle Replicas: 2 Resources: Limits: Cpu: 4100m Memory: 8697308774400m Requests: Cpu: 4100m Memory: 8697308774400m Service Version: 12.2.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 20Gi Name: fra Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 21Gi Component Def: oracle-observer-12c Name: observer Replicas: 1 Resources: Limits: Cpu: 500m Memory: 512Mi Requests: Cpu: 500m Memory: 512Mi Service Version: 12.2.0 Resources: Cpu: 0 Memory: 0 Storage: Size: 0 Termination Policy: Delete Topology: replication Status: Cluster Def Generation: 1 Components: Observer: Message: InstanceSet/oracle-hzwvnn-observer: ["oracle-hzwvnn-observer-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T04:44:42Z Oracle: Message: InstanceSet/oracle-hzwvnn-oracle: ["oracle-hzwvnn-oracle-0"] Phase: Running Pods Ready: true Pods Ready Time: 2025-09-01T04:59:29Z Conditions: Last Transition Time: 2025-09-01T03:19:20Z Message: The operator has started the provisioning of Cluster: oracle-hzwvnn Observed Generation: 9 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T04:52:06Z Message: Successfully applied for resources Observed Generation: 9 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T04:59:29Z Message: all pods of components are ready, waiting for the probe detection successful Reason: AllReplicasReady Status: True Type: ReplicasReady Last Transition Time: 2025-09-01T04:59:29Z Message: Cluster: oracle-hzwvnn is ready, current phase is Running Reason: ClusterReady Status: True Type: Ready Observed Generation: 9 Phase: Running Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning ReplicasNotReady 58m (x14 over 82m) cluster-controller pods are not ready in Components: [oracle], refer to related component message in Cluster.status.components Warning Abnormal 50m (x6 over 74m) cluster-controller Cluster: oracle-hzwvnn is Abnormal, check according to the components message Warning Unhealthy 36m (x22 over 55m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Database is not open, status: select open_mode from v$database * ERROR at line 1: ORA-01034: ORACLE not available Process ID: 0 Session ID: 0 Serial number: 0 Warning BackOff 31m (x5 over 49m) event-controller Pod oracle-hzwvnn-oracle-0: Back-off restarting failed container oracle in pod oracle-hzwvnn-oracle-0_ns-lrqxb(042ebe3d-c459-4efd-b59f-e33f1ae01b84) Normal ComponentPhaseTransition 31m (x8 over 74m) cluster-controller component is Failed Warning Unhealthy 18m (x15 over 20m) event-controller Pod oracle-hzwvnn-oracle-0: Readiness probe failed: Get "http://10.244.1.200:3501/v1.0/checkrole": context deadline exceeded (Client.Timeout exceeded while awaiting headers) Normal HorizontalScale 16m component-controller start horizontal scale component observer of cluster oracle-hzwvnn from 1 to 2 Normal HorizontalScale 16m component-controller start horizontal scale component observer of cluster oracle-hzwvnn from 2 to 1 Normal ComponentPhaseTransition 5m42s (x39 over 82m) cluster-controller component is Updating Normal ComponentPhaseTransition 52s (x33 over 83m) cluster-controller component is Running ------------------------------------------------------------------------------------------------------------------ check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover kill1 Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-oracle-hzwvnn" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-oracle-hzwvnn.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-oracle-hzwvnn created apply test-chaos-mesh-timeoffset-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-timeoffset-oracle-hzwvnn.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:58 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-oracle-hzwvnn" force deleted timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover timeoffset Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn created apply test-chaos-mesh-podfailure-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:58 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found check failover pod name failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn created apply test-chaos-mesh-podfailure-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:58 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn created apply test-chaos-mesh-podfailure-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:58 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn created apply test-chaos-mesh-podfailure-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:58 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn created apply test-chaos-mesh-podfailure-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:58 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found failover pod name:oracle-hzwvnn-oracle-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: pod-failure duration: 2m Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" not found `kubectl apply -f test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn created apply test-chaos-mesh-podfailure-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-podfailure-oracle-hzwvnn.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "oracle-hzwvnn-oracle-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 12:37 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:20 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-oracle-hzwvnn" force deleted podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-oracle-hzwvnn patched check failover pod name timeout `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success check component observer exists `kubectl get components -l app.kubernetes.io/instance=oracle-hzwvnn,apps.kubeblocks.io/component-name=observer --namespace ns-lrqxb | (grep "observer" || true )` cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart oracle-hzwvnn --auto-approve --force=true --components observer --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-restart-tnfsv created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-restart-tnfsv -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-restart-tnfsv ns-lrqxb Restart oracle-hzwvnn observer Running -/- Sep 01,2025 13:21 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:20 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-restart-tnfsv ns-lrqxb Restart oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 13:21 UTC+0800 check ops status done ops_status:oracle-hzwvnn-restart-tnfsv ns-lrqxb Restart oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 13:21 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-restart-tnfsv --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-restart-tnfsv patched `kbcli cluster delete-ops --name oracle-hzwvnn-restart-tnfsv --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-restart-tnfsv deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-oracle-hzwvnn" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-oracle-hzwvnn.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-oracle-hzwvnn created apply test-chaos-mesh-networkdelay-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-networkdelay-oracle-hzwvnn.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:20 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-oracle-hzwvnn" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover networkdelay Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-oracle-hzwvnn" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-oracle-hzwvnn.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-oracle-hzwvnn created apply test-chaos-mesh-dnsrandom-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-dnsrandom-oracle-hzwvnn.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Completed 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:20 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi pod_status:Completed pod_status:Completed pod_status:Completed pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff pod_status:CrashLoopBackOff check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-oracle-hzwvnn" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-oracle-hzwvnn" not found check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover dnsrandom Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover fullcpu check cluster status before cluster-failover-fullcpu cluster_status:Updating check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-oracle-hzwvnn" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-oracle-hzwvnn.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-oracle-hzwvnn created apply test-chaos-mesh-fullcpu-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-fullcpu-oracle-hzwvnn.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:20 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 12:52 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-oracle-hzwvnn" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-oracle-hzwvnn" not found check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover fullcpu Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart oracle-hzwvnn --auto-approve --force=true --components oracle --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-restart-mwcdf created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-restart-mwcdf -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-restart-mwcdf ns-lrqxb Restart oracle-hzwvnn oracle Pending -/- Sep 01,2025 13:30 UTC+0800 ops_status:oracle-hzwvnn-restart-mwcdf ns-lrqxb Restart oracle-hzwvnn oracle Creating -/- Sep 01,2025 13:30 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:31 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:30 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-restart-mwcdf ns-lrqxb Restart oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:30 UTC+0800 check ops status done ops_status:oracle-hzwvnn-restart-mwcdf ns-lrqxb Restart oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:30 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-restart-mwcdf --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-restart-mwcdf patched `kbcli cluster delete-ops --name oracle-hzwvnn-restart-mwcdf --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-restart-mwcdf deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster configure component_tmp: oracle check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure oracle-hzwvnn --auto-approve --force=true --set job_queue_processes=3999 --components oracle --config-spec oracle-config --config-file init.ora --namespace ns-lrqxb ` Will updated configure file meta: ConfigSpec: oracle-config ConfigFile: init.ora ComponentName: oracle ClusterName: oracle-hzwvnn OpsRequest oracle-hzwvnn-reconfiguring-wvbnm created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-reconfiguring-wvbnm -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-reconfiguring-wvbnm ns-lrqxb Reconfiguring oracle-hzwvnn oracle Pending -/- Sep 01,2025 13:33 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:31 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:20Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:30 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-reconfiguring-wvbnm ns-lrqxb Reconfiguring oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:33 UTC+0800 check ops status done ops_status:oracle-hzwvnn-reconfiguring-wvbnm ns-lrqxb Reconfiguring oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-reconfiguring-wvbnm --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-reconfiguring-wvbnm patched `kbcli cluster delete-ops --name oracle-hzwvnn-reconfiguring-wvbnm --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-reconfiguring-wvbnm deleted check config variables Defaulted container "oracle" out of: oracle, exporter, lorry, config-manager, oracle-init-container (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 3999 configure:[job_queue_processes] result actual:[3999] equal expected:[3999] `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success `kubectl get pvc -l app.kubernetes.io/instance=oracle-hzwvnn,apps.kubeblocks.io/component-name=oracle,apps.kubeblocks.io/vct-name=data --namespace ns-lrqxb ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in oracle-hzwvnn namespace. `kbcli cluster volume-expand oracle-hzwvnn --auto-approve --force=true --components oracle --volume-claim-templates data --storage 24Gi --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-volumeexpansion-2vc7w created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-volumeexpansion-2vc7w -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-volumeexpansion-2vc7w ns-lrqxb VolumeExpansion oracle-hzwvnn oracle Running -/- Sep 01,2025 13:33 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:31 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:30 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done No resources found in oracle-hzwvnn namespace. check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-volumeexpansion-2vc7w ns-lrqxb VolumeExpansion oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:33 UTC+0800 check ops status done ops_status:oracle-hzwvnn-volumeexpansion-2vc7w ns-lrqxb VolumeExpansion oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:33 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-volumeexpansion-2vc7w --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-volumeexpansion-2vc7w patched `kbcli cluster delete-ops --name oracle-hzwvnn-volumeexpansion-2vc7w --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-volumeexpansion-2vc7w deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster configure component_tmp: oracle check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster configure oracle-hzwvnn --auto-approve --force=true --set log_checkpoint_timeout=1801 --components oracle --config-spec oracle-config --config-file init.ora --namespace ns-lrqxb ` Will updated configure file meta: ConfigSpec: oracle-config ConfigFile: init.ora ComponentName: oracle ClusterName: oracle-hzwvnn OpsRequest oracle-hzwvnn-reconfiguring-sdk5r created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-reconfiguring-sdk5r -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-reconfiguring-sdk5r ns-lrqxb Reconfiguring oracle-hzwvnn oracle Pending -/- Sep 01,2025 13:39 UTC+0800 ops_status:oracle-hzwvnn-reconfiguring-sdk5r ns-lrqxb Reconfiguring oracle-hzwvnn oracle Creating -/- Sep 01,2025 13:39 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:22 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:40 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:39 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-reconfiguring-sdk5r ns-lrqxb Reconfiguring oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:39 UTC+0800 check ops status done ops_status:oracle-hzwvnn-reconfiguring-sdk5r ns-lrqxb Reconfiguring oracle-hzwvnn oracle Succeed 2/2 Sep 01,2025 13:39 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-reconfiguring-sdk5r --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-reconfiguring-sdk5r patched `kbcli cluster delete-ops --name oracle-hzwvnn-reconfiguring-sdk5r --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-reconfiguring-sdk5r deleted check config variables Defaulted container "oracle" out of: oracle, exporter, lorry, config-manager, oracle-init-container (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file current value_actual: 1801 configure:[log_checkpoint_timeout] result actual:[1801] equal expected:[1801] `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover connectionstress check cluster status before cluster-failover-connectionstress check cluster status done cluster_status:Running check node drain check node drain success Error from server (NotFound): pods "test-db-client-connectionstress-oracle-hzwvnn" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-oracle-hzwvnn --namespace ns-lrqxb ` Error from server (NotFound): pods "test-db-client-connectionstress-oracle-hzwvnn" not found Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): pods "test-db-client-connectionstress-oracle-hzwvnn" not found `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: apiVersion: v1 kind: Pod metadata: name: test-db-client-connectionstress-oracle-hzwvnn namespace: ns-lrqxb spec: containers: - name: test-dbclient imagePullPolicy: IfNotPresent image: docker.io/apecloud/dbclient:test args: - "--host" - "oracle-hzwvnn-oracle.ns-lrqxb.svc.cluster.local" - "--user" - "sys" - "--password" - "U970u#h1N76u" - "--port" - "1521" - "--database" - "" - "--dbtype" - "oracle" - "--test" - "connectionstress" - "--connections" - "640" - "--duration" - "60" restartPolicy: Never `kubectl apply -f test-db-client-connectionstress-oracle-hzwvnn.yaml` pod/test-db-client-connectionstress-oracle-hzwvnn created apply test-db-client-connectionstress-oracle-hzwvnn.yaml Success `rm -rf test-db-client-connectionstress-oracle-hzwvnn.yaml` check pod status pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-oracle-hzwvnn 1/1 Running 0 6s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-oracle-hzwvnn 1/1 Running 0 10s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-oracle-hzwvnn 1/1 Running 0 15s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-oracle-hzwvnn 1/1 Running 0 20s pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-oracle-hzwvnn 1/1 Running 0 25s check pod test-db-client-connectionstress-oracle-hzwvnn status done pod_status:NAME READY STATUS RESTARTS AGE test-db-client-connectionstress-oracle-hzwvnn 0/1 Completed 0 30s check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Abnormal Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "oracle-hzwvnn-observer-0" force deleted cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 500m / 500m 512Mi / 512Mi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:47 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:40 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:39 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:557) at oracle.net.ns.NIOPacket.readNIOPacket(NIOPacket.java:403) at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:127) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:340) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1596) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:588) ... 11 more Failed to connect to Oracle database: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call Trying with database Oracle. java.io.IOException: Failed to connect to Oracle database: at com.apecloud.dbtester.tester.OracleTester.connect(OracleTester.java:73) at com.apecloud.dbtester.tester.OracleTester.connectionStress(OracleTester.java:121) at com.apecloud.dbtester.commons.TestExecutor.executeTest(TestExecutor.java:33) at OneClient.executeTest(OneClient.java:108) at OneClient.main(OneClient.java:40) Caused by: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:854) at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:793) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:57) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:562) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.OracleTester.connect(OracleTester.java:71) ... 4 more Caused by: oracle.net.ns.NetException: Got minus one from a read call at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:557) at oracle.net.ns.NIOPacket.readNIOPacket(NIOPacket.java:403) at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:127) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:340) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1596) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:588) ... 11 more Failed to connect to Oracle database: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call Trying with database Oracle. java.io.IOException: Failed to connect to Oracle database: at com.apecloud.dbtester.tester.OracleTester.connect(OracleTester.java:73) at com.apecloud.dbtester.tester.OracleTester.connectionStress(OracleTester.java:121) at com.apecloud.dbtester.commons.TestExecutor.executeTest(TestExecutor.java:33) at OneClient.executeTest(OneClient.java:108) at OneClient.main(OneClient.java:40) Caused by: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:854) at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:793) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:57) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:562) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.OracleTester.connect(OracleTester.java:71) ... 4 more Caused by: oracle.net.ns.NetException: Got minus one from a read call at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:557) at oracle.net.ns.NIOPacket.readNIOPacket(NIOPacket.java:403) at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:127) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:340) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1596) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:588) ... 11 more Failed to connect to Oracle database: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call Trying with database Oracle. java.io.IOException: Failed to connect to Oracle database: at com.apecloud.dbtester.tester.OracleTester.connect(OracleTester.java:73) at com.apecloud.dbtester.tester.OracleTester.connectionStress(OracleTester.java:121) at com.apecloud.dbtester.commons.TestExecutor.executeTest(TestExecutor.java:33) at OneClient.executeTest(OneClient.java:108) at OneClient.main(OneClient.java:40) Caused by: java.sql.SQLRecoverableException: IO Error: Got minus one from a read call at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:854) at oracle.jdbc.driver.PhysicalConnection.connect(PhysicalConnection.java:793) at oracle.jdbc.driver.T4CDriverExtension.getConnection(T4CDriverExtension.java:57) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:747) at oracle.jdbc.driver.OracleDriver.connect(OracleDriver.java:562) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at java.sql/java.sql.DriverManager.getConnection(Unknown Source) at com.apecloud.dbtester.tester.OracleTester.connect(OracleTester.java:71) ... 4 more Caused by: oracle.net.ns.NetException: Got minus one from a read call at oracle.net.ns.NSProtocolNIO.doSocketRead(NSProtocolNIO.java:557) at oracle.net.ns.NIOPacket.readNIOPacket(NIOPacket.java:403) at oracle.net.ns.NSProtocolNIO.negotiateConnection(NSProtocolNIO.java:127) at oracle.net.ns.NSProtocol.connect(NSProtocol.java:340) at oracle.jdbc.driver.T4CConnection.connect(T4CConnection.java:1596) at oracle.jdbc.driver.T4CConnection.logon(T4CConnection.java:588) ... 11 more Test Result: null Connection Information: Database Type: oracle Host: oracle-hzwvnn-oracle.ns-lrqxb.svc.cluster.local Port: 1521 Database: Table: User: sys Org: Access Mode: mysql Test Type: connectionstress Connection Count: 640 Duration: 60 seconds `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge pods test-db-client-connectionstress-oracle-hzwvnn --namespace ns-lrqxb ` pod/test-db-client-connectionstress-oracle-hzwvnn patched (no change) Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "test-db-client-connectionstress-oracle-hzwvnn" force deleted check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover connectionstress Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success check component observer exists `kubectl get components -l app.kubernetes.io/instance=oracle-hzwvnn,apps.kubeblocks.io/component-name=observer --namespace ns-lrqxb | (grep "observer" || true )` cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale oracle-hzwvnn --auto-approve --force=true --components observer --cpu 4100m --memory 8.1Gi --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-verticalscaling-bwrs5 created successfully, you can view the progress: kbcli cluster describe-ops oracle-hzwvnn-verticalscaling-bwrs5 -n ns-lrqxb check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-verticalscaling-bwrs5 ns-lrqxb VerticalScaling oracle-hzwvnn observer Creating -/- Sep 01,2025 13:47 UTC+0800 check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 4100m / 4100m 8697308774400m / 8697308774400m aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:48 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:40 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:39 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done check ops status `kbcli cluster list-ops oracle-hzwvnn --status all --namespace ns-lrqxb ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME oracle-hzwvnn-verticalscaling-bwrs5 ns-lrqxb VerticalScaling oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 13:47 UTC+0800 check ops status done ops_status:oracle-hzwvnn-verticalscaling-bwrs5 ns-lrqxb VerticalScaling oracle-hzwvnn observer Succeed 1/1 Sep 01,2025 13:47 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests oracle-hzwvnn-verticalscaling-bwrs5 --namespace ns-lrqxb ` opsrequest.apps.kubeblocks.io/oracle-hzwvnn-verticalscaling-bwrs5 patched `kbcli cluster delete-ops --name oracle-hzwvnn-verticalscaling-bwrs5 --force --auto-approve --namespace ns-lrqxb ` OpsRequest oracle-hzwvnn-verticalscaling-bwrs5 deleted `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-oracle-hzwvnn" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-oracle-hzwvnn.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-oracle-hzwvnn created apply test-chaos-mesh-networkduplicate-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-networkduplicate-oracle-hzwvnn.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 4100m / 4100m 8697308774400m / 8697308774400m aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:48 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:40 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:39 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-oracle-hzwvnn" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-oracle-hzwvnn" not found check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover networkduplicate Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-oracle-hzwvnn" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-oracle-hzwvnn" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-oracle-hzwvnn namespace: ns-lrqxb spec: selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-0 action: partition mode: all target: mode: all selector: namespaces: - ns-lrqxb labelSelectors: apps.kubeblocks.io/pod-name: oracle-hzwvnn-oracle-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-oracle-hzwvnn.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-oracle-hzwvnn created apply test-chaos-mesh-networkpartition-oracle-hzwvnn.yaml Success `rm -rf test-chaos-mesh-networkpartition-oracle-hzwvnn.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle Delete Running Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 4100m / 4100m 8697308774400m / 8697308774400m aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:48 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:40 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:39 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-oracle-hzwvnn --namespace ns-lrqxb ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-oracle-hzwvnn" force deleted networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-oracle-hzwvnn patched check failover pod name failover pod name:oracle-hzwvnn-oracle-0 failover networkpartition Success `kubectl get secrets -l app.kubernetes.io/instance=oracle-hzwvnn` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.username***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.password***"` `kubectl get secrets oracle-hzwvnn-oracle-account-sys -o jsonpath="***.data.port***"` DB_USERNAME:sys;DB_PASSWORD:U970u#h1N76u;DB_PORT:1521;DB_DATABASE: check db_client batch data count `echo "echo \"select count(*) from sys.executions_loop_table;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check db_client batch data Success cluster update terminationPolicy WipeOut `kbcli cluster update oracle-hzwvnn --termination-policy=WipeOut --namespace ns-lrqxb ` cluster.apps.kubeblocks.io/oracle-hzwvnn updated check cluster status `kbcli cluster list oracle-hzwvnn --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn ns-lrqxb oracle WipeOut Updating Sep 01,2025 11:19 UTC+0800 app.kubernetes.io/instance=oracle-hzwvnn,clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-observer-0 ns-lrqxb oracle-hzwvnn observer Running 0 4100m / 4100m 8697308774400m / 8697308774400m aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:48 UTC+0800 oracle-hzwvnn-oracle-0 ns-lrqxb oracle-hzwvnn oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:40 UTC+0800 fra:21Gi oracle-hzwvnn-oracle-1 ns-lrqxb oracle-hzwvnn oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:39 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-oracle-0;secondary: oracle-hzwvnn-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done cluster oracle-rman backup `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.name***"` `kubectl get backuprepo backuprepo-kbcli-test -o jsonpath="***.spec.credential.namespace***"` `kubectl get secrets kb-backuprepo-vwx9l -n kb-cxqgy -o jsonpath="***.data.accessKeyId***"` `kubectl get secrets kb-backuprepo-vwx9l -n kb-cxqgy -o jsonpath="***.data.secretAccessKey***"` KUBEBLOCKS NAMESPACE:kb-cxqgy get kubeblocks namespace done `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-cxqgy -o jsonpath="***.items[0].data.root-user***"` `kubectl get secrets -l app.kubernetes.io/instance=kbcli-test-minio --namespace kb-cxqgy -o jsonpath="***.items[0].data.root-password***"` minio_user:kbclitest,minio_password:kbclitest,minio_endpoint:kbcli-test-minio.kb-cxqgy.svc.cluster.local:9000 list minio bucket kbcli-test `echo 'mc config host add minioserver http://kbcli-test-minio.kb-cxqgy.svc.cluster.local:9000 kbclitest kbclitest;mc ls minioserver' | kubectl exec -it kbcli-test-minio-fd76fc655-fcvnb --namespace kb-cxqgy -- bash` Unable to use a TTY - input is not a terminal or the right kind of file list minio bucket done default backuprepo:backuprepo-kbcli-test exists `kbcli cluster backup oracle-hzwvnn --method oracle-rman --namespace ns-lrqxb ` Backup backup-ns-lrqxb-oracle-hzwvnn-20250901135421 created successfully, you can view the progress: kbcli cluster list-backups --name=backup-ns-lrqxb-oracle-hzwvnn-20250901135421 -n ns-lrqxb check backup status `kbcli cluster list-backups oracle-hzwvnn --namespace ns-lrqxb ` No backups found in ns-lrqxb namespace. backup_status:oracle-hzwvnn-oracle-rman-Sep backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running backup_status:oracle-hzwvnn-oracle-rman-Running check backup status done backup_status:backup-ns-lrqxb-oracle-hzwvnn-20250901135421 ns-lrqxb oracle-hzwvnn oracle-rman Completed 654830117 2m57s Sep 01,2025 13:54 UTC+0800 Sep 01,2025 13:57 UTC+0800 cluster restore backup Error from server (NotFound): opsrequests.apps.kubeblocks.io "oracle-hzwvnn-backup" not found `kbcli cluster describe-backup backup-ns-lrqxb-oracle-hzwvnn-20250901135421 --namespace ns-lrqxb ` Name: backup-ns-lrqxb-oracle-hzwvnn-20250901135421 Cluster: oracle-hzwvnn Namespace: ns-lrqxb Spec: Method: oracle-rman Policy Name: oracle-hzwvnn-oracle-backup-policy Status: Phase: Completed Total Size: 654830117 ActionSet Name: oracle-rman Repository: backuprepo-kbcli-test Duration: 2m57s Start Time: Sep 01,2025 13:54 UTC+0800 Completion Time: Sep 01,2025 13:57 UTC+0800 Path: /ns-lrqxb/oracle-hzwvnn-302b65fe-03dc-4b9a-8045-6498e180e60d/oracle/backup-ns-lrqxb-oracle-hzwvnn-20250901135421 Warning Events: `kbcli cluster restore oracle-hzwvnn-backup --backup backup-ns-lrqxb-oracle-hzwvnn-20250901135421 --namespace ns-lrqxb ` Cluster oracle-hzwvnn-backup created check cluster status `kbcli cluster list oracle-hzwvnn-backup --show-labels --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS oracle-hzwvnn-backup ns-lrqxb oracle WipeOut Sep 01,2025 13:57 UTC+0800 clusterdefinition.kubeblocks.io/name=oracle,clusterversion.kubeblocks.io/name= cluster_status: cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances oracle-hzwvnn-backup --namespace ns-lrqxb ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME oracle-hzwvnn-backup-observer-0 ns-lrqxb oracle-hzwvnn-backup observer Running 0 4100m / 4100m 8697308774400m / 8697308774400m aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 14:06 UTC+0800 oracle-hzwvnn-backup-oracle-0 ns-lrqxb oracle-hzwvnn-backup oracle Running primary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000005/10.224.0.6 Sep 01,2025 13:58 UTC+0800 fra:21Gi oracle-hzwvnn-backup-oracle-1 ns-lrqxb oracle-hzwvnn-backup oracle Running secondary 0 4100m / 4100m 8697308774400m / 8697308774400m data:24Gi aks-cicdamdpool-15164480-vmss000000/10.224.0.5 Sep 01,2025 13:58 UTC+0800 fra:21Gi check pod status done check cluster role check cluster role done primary: oracle-hzwvnn-backup-oracle-0;secondary: oracle-hzwvnn-backup-oracle-1 check cluster connect `echo "echo \"\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-backup-oracle-0 --namespace ns-lrqxb -- bash ` check cluster connect done `kbcli cluster describe-backup backup-ns-lrqxb-oracle-hzwvnn-20250901135421 --namespace ns-lrqxb ` Name: backup-ns-lrqxb-oracle-hzwvnn-20250901135421 Cluster: oracle-hzwvnn Namespace: ns-lrqxb Spec: Method: oracle-rman Policy Name: oracle-hzwvnn-oracle-backup-policy Status: Phase: Completed Total Size: 654830117 ActionSet Name: oracle-rman Repository: backuprepo-kbcli-test Duration: 2m57s Start Time: Sep 01,2025 13:54 UTC+0800 Completion Time: Sep 01,2025 13:57 UTC+0800 Path: /ns-lrqxb/oracle-hzwvnn-302b65fe-03dc-4b9a-8045-6498e180e60d/oracle/backup-ns-lrqxb-oracle-hzwvnn-20250901135421 Warning Events: cluster connect `echo "echo \"select INSTANCE_NUMBER,INSTANCE_NAME,HOST_NAME,STATUS,DATABASE_STATUS,INSTANCE_ROLE,DATABASE_TYPE from v\\$instance;\" | sqlplus -S / as sysdba " | kubectl exec -it oracle-hzwvnn-backup-oracle-0 --namespace ns-lrqxb -- bash ` Defaulted container "oracle" out of: oracle, exporter, lorry, config-manager, oracle-init-container (init), init-lorry (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file INSTANCE_NUMBER INSTANCE_NAME --------------- ---------------- HOST_NAME STATUS ---------------------------------------------------------------- ------------ DATABASE_STATUS INSTANCE_ROLE DATABASE_TYPE ----------------- ------------------ --------------- 1 ORCLCDB oracle-hzwvnn-backup-oracle-0 OPEN ACTIVE PRIMARY_INSTANCE SINGLE connect cluster Success delete cluster oracle-hzwvnn-backup `kbcli cluster delete oracle-hzwvnn-backup --auto-approve --namespace ns-lrqxb ` Cluster oracle-hzwvnn-backup deleted pod_info:oracle-hzwvnn-backup-observer-0 1/1 Running 0 23s oracle-hzwvnn-backup-oracle-0 4/4 Running 0 8m53s oracle-hzwvnn-backup-oracle-1 4/4 Running 0 8m53s pod_info:oracle-hzwvnn-backup-observer-0 1/1 Terminating 0 43s oracle-hzwvnn-backup-oracle-0 4/4 Running 0 9m13s oracle-hzwvnn-backup-oracle-1 4/4 Running 0 9m13s No resources found in ns-lrqxb namespace. delete cluster pod done No resources found in ns-lrqxb namespace. check cluster resource non-exist OK: pvc No resources found in ns-lrqxb namespace. delete cluster done No resources found in ns-lrqxb namespace. No resources found in ns-lrqxb namespace. No resources found in ns-lrqxb namespace. cluster delete backup `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge backups backup-ns-lrqxb-oracle-hzwvnn-20250901135421 --namespace ns-lrqxb ` backup.dataprotection.kubeblocks.io/backup-ns-lrqxb-oracle-hzwvnn-20250901135421 patched `kbcli cluster delete-backup oracle-hzwvnn --name backup-ns-lrqxb-oracle-hzwvnn-20250901135421 --force --auto-approve --namespace ns-lrqxb ` Backup backup-ns-lrqxb-oracle-hzwvnn-20250901135421 deleted No opsrequests found in ns-lrqxb namespace. cluster list-logs `kbcli cluster list-logs oracle-hzwvnn --namespace ns-lrqxb ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update oracle-hzwvnn --enable-all-logs=true --namespace ns-lrqxb Error from server (NotFound): pods "oracle-hzwvnn-oracle-0" not found cluster logs `kbcli cluster logs oracle-hzwvnn --tail 30 --namespace ns-lrqxb ` Defaulted container "oracle" out of: oracle, exporter, lorry, config-manager, oracle-init-container (init), init-lorry (init), install-config-manager-tool (init) Current log# 2 seq# 65 mem# 0: /opt/oracle/oradata/ORCLCDB_0/redo02.log 2025-09-01T05:55:26.207380+00:00 Archived Log entry 121 added for T-1.S-64 ID 0xb1b96e5a LAD:1 2025-09-01T05:55:27.385690+00:00 Starting control autobackup Control autobackup written to DISK device handle '/opt/oracle/flash_recovery_area/ORCLCDB_0/autobackup/2025_09_01/o1_mf_s_1210658127_ncbfbhnt_.bkp' 2025-09-01T05:55:31.086733+00:00 Starting control autobackup Control autobackup written to DISK device handle '/opt/oracle/flash_recovery_area/ORCLCDB_0/autobackup/2025_09_01/o1_mf_s_1210658131_ncbfbm7h_.bkp' 2025-09-01T05:55:33.545745+00:00 Starting control autobackup Control autobackup written to DISK device handle '/opt/oracle/flash_recovery_area/ORCLCDB_0/autobackup/2025_09_01/o1_mf_s_1210658133_ncbfbooh_.bkp' 2025-09-01T05:59:10.414095+00:00 TT05: Standby redo logfile selected for thread 1 sequence 65 for destination LOG_ARCHIVE_DEST_2 2025-09-01T06:00:10.535977+00:00 ARC6: Standby redo logfile selected for thread 1 sequence 63 for destination LOG_ARCHIVE_DEST_2 2025-09-01T06:01:07.356614+00:00 Resize operation completed for file# 3, old size 481280K, new size 491520K delete cluster oracle-hzwvnn `kbcli cluster delete oracle-hzwvnn --auto-approve --namespace ns-lrqxb ` Cluster oracle-hzwvnn deleted pod_info:oracle-hzwvnn-observer-0 0/1 CrashLoopBackOff 7 (4m39s ago) 19m oracle-hzwvnn-oracle-0 4/4 Running 0 27m oracle-hzwvnn-oracle-1 4/4 Running 0 28m No resources found in ns-lrqxb namespace. delete cluster pod done No resources found in ns-lrqxb namespace. check cluster resource non-exist OK: pvc No resources found in ns-lrqxb namespace. delete cluster done No resources found in ns-lrqxb namespace. No resources found in ns-lrqxb namespace. No resources found in ns-lrqxb namespace. Oracle Test Suite All Done! --------------------------------------Oracle (Topology = replication Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[ClusterDefinition=oracle;]|[Description=Create a cluster with the specified cluster definition oracle] [PASSED]|[Connect]|[ComponentName=oracle]|[Description=Connect to the cluster] [SKIPPED]|[-]|[HA=Evicting Pod;ComponentName=oracle]|[Description=-] [WARNING]|[CheckFailover]|[FailoverType=podkill]|[Description=-] [PASSED]|[No-Failover]|[HA=Pod Kill;ComponentName=oracle]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [FAILED]|[SwitchOver]|[ComponentName=oracle]|[Description=SwitchOver the cluster specify component oracle] [WARNING]|[CheckFailover]|[FailoverType=oom]|[Description=-] [FAILED]|[No-Failover]|[HA=OOM;Durations=2m;ComponentName=oracle]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=oracle]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[No-Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=oracle]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[No-Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=oracle]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[HorizontalScaling Out]|[ComponentName=observer]|[Description=HorizontalScaling Out the cluster specify component observer] [PASSED]|[HorizontalScaling In]|[ComponentName=observer]|[Description=HorizontalScaling In the cluster specify component observer] [PASSED]|[No-Failover]|[HA=Network Loss;Durations=2m;ComponentName=oracle]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[VolumeExpansion]|[ComponentName=oracle]|[Description=VolumeExpansion the cluster specify component oracle] [PASSED]|[VerticalScaling]|[ComponentName=oracle]|[Description=VerticalScaling the cluster specify component oracle] [FAILED]|[Check Connect]|[-]|[Description=-] [FAILED]|[No-Failover]|[HA=Kill 1;ComponentName=oracle]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=oracle]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [WARNING]|[CheckFailover]|[FailoverType=podfailure]|[Description=-] [PASSED]|[No-Failover]|[HA=Pod Failure;Durations=2m;ComponentName=oracle]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[Restart]|[ComponentName=observer]|[Description=Restart the cluster specify component observer] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=oracle]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=oracle]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=oracle]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[Restart]|[ComponentName=oracle]|[Description=Restart the cluster specify component oracle] [PASSED]|[Reconfiguring]|[ComponentName=oracle;job_queue_processes=3999]|[Description=Reconfiguring the cluster specify component oracle set job_queue_processes=3999] [PASSED]|[VolumeExpansion]|[ComponentName=oracle]|[Description=VolumeExpansion the cluster specify component oracle] [PASSED]|[Reconfiguring]|[ComponentName=oracle;log_checkpoint_timeout=1801]|[Description=Reconfiguring the cluster specify component oracle set log_checkpoint_timeout=1801] [PASSED]|[No-Failover]|[HA=Connection Stress;ComponentName=oracle]|[Description=Simulates conditions where pods experience connection stress either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Connection load.] [PASSED]|[VerticalScaling]|[ComponentName=observer]|[Description=VerticalScaling the cluster specify component observer] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=oracle]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=oracle]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Backup]|[BackupMethod=oracle-rman]|[Description=The cluster oracle-rman Backup] [PASSED]|[Restore]|[BackupMethod=oracle-rman]|[Description=The cluster oracle-rman Restore] [PASSED]|[Connect]|[ComponentName=oracle]|[Description=Connect to the cluster] [PASSED]|[Delete Restore Cluster]|[BackupMethod=oracle-rman]|[Description=Delete the oracle-rman restore cluster] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]