bash test/kbcli/test_kbcli_0.9.sh --type 20 --version 0.9.5 --generate-output true --chaos-mesh true --drain-node true --random-namespace true --region eastus --cloud-provider aks CURRENT_TEST_DIR:test/kbcli source commons files source engines files source kubeblocks files `kubectl get namespace | grep ns-drhff ` `kubectl create namespace ns-drhff` namespace/ns-drhff created create namespace ns-drhff done download kbcli `gh release list --repo apecloud/kbcli --limit 100 | (grep "0.9" || true)` `curl -fsSL https://kubeblocks.io/installer/install_cli.sh | bash -s v0.9.5-beta.8` Your system is linux_amd64 Installing kbcli ... Downloading ... % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0 100 32.1M 100 32.1M 0 0 123M 0 --:--:-- --:--:-- --:--:-- 123M kbcli installed successfully. Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Make sure your docker service is running and begin your journey with kbcli: kbcli playground init For more information on how to get started, please visit: https://kubeblocks.io download kbcli v0.9.5-beta.8 done Kubernetes: v1.32.6 KubeBlocks: 0.9.5 kbcli: 0.9.5-beta.8 WARNING: version difference between kbcli (0.9.5-beta.8) and kubeblocks (0.9.5) Kubernetes Env: v1.32.6 POD_RESOURCES: aks kb-default-sc found aks default-vsc found found default storage class: default kubeblocks version is:0.9.5 skip upgrade kubeblocks Error: no repositories to show helm repo add chaos-mesh https://charts.chaos-mesh.org "chaos-mesh" has been added to your repositories add helm chart repo chaos-mesh success chaos mesh already installed check cluster definition set component name:postgresql set component version set component version:vanilla-postgresql set service versions:12.15.0,14.7.0,15.6.1138,15.7.0 set service versions sorted:12.15.0,14.7.0,15.6.1138,15.7.0 no cluster version found unsupported component definition REPORT_COUNT 0:0 set replicas first:2,12.15.0|2,14.7.0|2,15.6.1138|2,15.7.0 set replicas third:2,12.15.0 set replicas fourth:2,12.15.0 set minimum cmpv service version set minimum cmpv service version replicas:2,12.15.0 REPORT_COUNT:1 CLUSTER_TOPOLOGY: set cluster topology: vanilla-postgresql LIMIT_CPU:0.5 LIMIT_MEMORY:0.5 storage size: 3 No resources found in ns-drhff namespace. termination_policy:Delete create 2 replica Delete vanilla-postgresql cluster check cluster definition check component definition set component definition by component version check cmpd by labels set component definition1: vanilla-postgresql-12 by component version:vanilla-postgresql apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: name: vanilla-ewmtgg namespace: ns-drhff spec: clusterDefinitionRef: vanilla-postgresql topology: vanilla-postgresql terminationPolicy: Delete componentSpecs: - name: postgresql serviceVersion: 12.15.0 replicas: 2 resources: requests: cpu: 500m memory: 0.5Gi limits: cpu: 500m memory: 0.5Gi volumeClaimTemplates: - name: data spec: storageClassName: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi `kubectl apply -f test_create_vanilla-ewmtgg.yaml` cluster.apps.kubeblocks.io/vanilla-ewmtgg created apply test_create_vanilla-ewmtgg.yaml Success `rm -rf test_create_vanilla-ewmtgg.yaml` check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Sep 01,2025 11:17 UTC+0800 clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status: cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating cluster_status:Creating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 500m / 500m 512Mi / 512Mi data:3Gi aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 500m / 500m 512Mi / 512Mi data:3Gi aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl get secrets -l app.kubernetes.io/instance=vanilla-ewmtgg` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:D4557qD3Na;DB_PORT:5432;DB_DATABASE: check pod vanilla-ewmtgg-postgresql-0 container_name postgresql exist password D4557qD3Na check pod vanilla-ewmtgg-postgresql-0 container_name lorry exist password D4557qD3Na check pod vanilla-ewmtgg-postgresql-0 container_name config-manager exist password D4557qD3Na No container logs contain secret password. describe cluster `kbcli cluster describe vanilla-ewmtgg --namespace ns-drhff ` Name: vanilla-ewmtgg Created Time: Sep 01,2025 11:17 UTC+0800 NAMESPACE CLUSTER-DEFINITION VERSION STATUS TERMINATION-POLICY ns-drhff vanilla-postgresql Running Delete Endpoints: COMPONENT MODE INTERNAL EXTERNAL postgresql ReadWrite vanilla-ewmtgg-postgresql.ns-drhff.svc.cluster.local:5432 Topology: COMPONENT INSTANCE ROLE STATUS AZ NODE CREATED-TIME postgresql vanilla-ewmtgg-postgresql-0 primary Running 0 aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 postgresql vanilla-ewmtgg-postgresql-1 secondary Running 0 aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 Resources Allocation: COMPONENT DEDICATED CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE-SIZE STORAGE-CLASS postgresql false 500m / 500m 512Mi / 512Mi data:3Gi default Images: COMPONENT TYPE IMAGE postgresql docker.io/apecloud/postgres:12.15 Data Protection: BACKUP-REPO AUTO-BACKUP BACKUP-SCHEDULE BACKUP-METHOD BACKUP-RETENTION RECOVERABLE-TIME Show cluster events: kbcli cluster list-events -n ns-drhff vanilla-ewmtgg `kbcli cluster label vanilla-ewmtgg app.kubernetes.io/instance- --namespace ns-drhff ` label "app.kubernetes.io/instance" not found. `kbcli cluster label vanilla-ewmtgg app.kubernetes.io/instance=vanilla-ewmtgg --namespace ns-drhff ` `kbcli cluster label vanilla-ewmtgg --list --namespace ns-drhff ` NAME NAMESPACE LABELS vanilla-ewmtgg ns-drhff app.kubernetes.io/instance=vanilla-ewmtgg clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= label cluster app.kubernetes.io/instance=vanilla-ewmtgg Success `kbcli cluster label case.name=kbcli.test1 -l app.kubernetes.io/instance=vanilla-ewmtgg --namespace ns-drhff ` `kbcli cluster label vanilla-ewmtgg --list --namespace ns-drhff ` NAME NAMESPACE LABELS vanilla-ewmtgg ns-drhff app.kubernetes.io/instance=vanilla-ewmtgg case.name=kbcli.test1 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test1 Success `kbcli cluster label vanilla-ewmtgg case.name=kbcli.test2 --overwrite --namespace ns-drhff ` `kbcli cluster label vanilla-ewmtgg --list --namespace ns-drhff ` NAME NAMESPACE LABELS vanilla-ewmtgg ns-drhff app.kubernetes.io/instance=vanilla-ewmtgg case.name=kbcli.test2 clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= label cluster case.name=kbcli.test2 Success `kbcli cluster label vanilla-ewmtgg case.name- --namespace ns-drhff ` `kbcli cluster label vanilla-ewmtgg --list --namespace ns-drhff ` NAME NAMESPACE LABELS vanilla-ewmtgg ns-drhff app.kubernetes.io/instance=vanilla-ewmtgg clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= delete cluster label case.name Success cluster connect `echo 'select 1' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff ` Unable to use a TTY - input is not a terminal or the right kind of file Connect to instance vanilla-ewmtgg-postgresql-0: out of vanilla-ewmtgg-postgresql-0(primary), vanilla-ewmtgg-postgresql-1(secondary) ?column? ---------- 1 (1 row) connect cluster Success cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose vanilla-ewmtgg --auto-approve --force=true --type vpc --enable true --components postgresql --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-expose-29ln7 created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-expose-29ln7 -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 500m / 500m 512Mi / 512Mi data:3Gi aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 500m / 500m 512Mi / 512Mi data:3Gi aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 11:20 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-expose-29ln7 ns-drhff Expose vanilla-ewmtgg postgresql Succeed 1/1 Sep 01,2025 11:20 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-expose-29ln7 --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-expose-29ln7 patched `kbcli cluster delete-ops --name vanilla-ewmtgg-expose-29ln7 --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-expose-29ln7 deleted test failover podfailure check cluster status before cluster-failover-podfailure check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-ewmtgg" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podfailure-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all action: pod-failure duration: 2m `kubectl apply -f test-chaos-mesh-podfailure-vanilla-ewmtgg.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podfailure-vanilla-ewmtgg created apply test-chaos-mesh-podfailure-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-podfailure-vanilla-ewmtgg.yaml` podfailure chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Abnormal Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 500m / 500m 512Mi / 512Mi data:3Gi aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 500m / 500m 512Mi / 512Mi data:3Gi aks-cicdamdpool-68727890-vmss000001/10.224.0.7 Sep 01,2025 11:17 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podfailure-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-ewmtgg" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podfailure-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover podfailure Success cluster vscale check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster vscale vanilla-ewmtgg --auto-approve --force=true --components postgresql --cpu 600m --memory 0.6Gi --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-verticalscaling-tr9ft created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-verticalscaling-tr9ft -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-verticalscaling-tr9ft ns-drhff VerticalScaling vanilla-ewmtgg postgresql Running 0/2 Sep 01,2025 11:24 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating [Error] check cluster status timeout --------------------------------------get cluster vanilla-ewmtgg yaml-------------------------------------- `kubectl get cluster vanilla-ewmtgg -o yaml --namespace ns-drhff ` apiVersion: apps.kubeblocks.io/v1alpha1 kind: Cluster metadata: annotations: kubeblocks.io/ops-request: '[***"name":"vanilla-ewmtgg-verticalscaling-tr9ft","type":"VerticalScaling"***]' kubectl.kubernetes.io/last-applied-configuration: | ***"apiVersion":"apps.kubeblocks.io/v1alpha1","kind":"Cluster","metadata":***"annotations":***,"name":"vanilla-ewmtgg","namespace":"ns-drhff"***,"spec":***"clusterDefinitionRef":"vanilla-postgresql","componentSpecs":[***"name":"postgresql","replicas":2,"resources":***"limits":***"cpu":"500m","memory":"0.5Gi"***,"requests":***"cpu":"500m","memory":"0.5Gi"***,"serviceVersion":"12.15.0","volumeClaimTemplates":[***"name":"data","spec":***"accessModes":["ReadWriteOnce"],"resources":***"requests":***"storage":"3Gi"***,"storageClassName":null***]***],"terminationPolicy":"Delete","topology":"vanilla-postgresql"*** creationTimestamp: "2025-09-01T03:17:21Z" finalizers: - cluster.kubeblocks.io/finalizer generation: 6 labels: app.kubernetes.io/instance: vanilla-ewmtgg clusterdefinition.kubeblocks.io/name: vanilla-postgresql clusterversion.kubeblocks.io/name: "" name: vanilla-ewmtgg namespace: ns-drhff resourceVersion: "22781" uid: afc63470-0298-4a0e-afa4-2743e92970fc spec: clusterDefinitionRef: vanilla-postgresql componentSpecs: - componentDef: vanilla-postgresql-12 name: postgresql replicas: 2 resources: limits: cpu: 600m memory: 644245094400m requests: cpu: 600m memory: 644245094400m serviceVersion: 12.15.0 volumeClaimTemplates: - name: data spec: accessModes: - ReadWriteOnce resources: requests: storage: 3Gi resources: cpu: "0" memory: "0" services: - annotations: service.beta.kubernetes.io/azure-load-balancer-internal: "true" componentSelector: postgresql name: postgresql-vpc roleSelector: primary serviceName: postgresql-vpc spec: ports: - name: tcp-postgresql nodePort: 30402 port: 5432 protocol: TCP targetPort: tcp-postgresql type: LoadBalancer storage: size: "0" terminationPolicy: Delete topology: vanilla-postgresql status: clusterDefGeneration: 1 components: postgresql: phase: Updating podsReady: false podsReadyTime: "2025-09-01T03:24:34Z" conditions: - lastTransitionTime: "2025-09-01T03:17:21Z" message: 'The operator has started the provisioning of Cluster: vanilla-ewmtgg' observedGeneration: 6 reason: PreCheckSucceed status: "True" type: ProvisioningStarted - lastTransitionTime: "2025-09-01T03:17:21Z" message: Successfully applied for resources observedGeneration: 6 reason: ApplyResourcesSucceed status: "True" type: ApplyResources - lastTransitionTime: "2025-09-01T03:24:46Z" message: 'pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ReplicasNotReady status: "False" type: ReplicasReady - lastTransitionTime: "2025-09-01T03:24:46Z" message: 'pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components' reason: ComponentsNotReady status: "False" type: Ready observedGeneration: 6 phase: Updating ------------------------------------------------------------------------------------------------------------------ --------------------------------------describe cluster vanilla-ewmtgg-------------------------------------- `kubectl describe cluster vanilla-ewmtgg --namespace ns-drhff ` Name: vanilla-ewmtgg Namespace: ns-drhff Labels: app.kubernetes.io/instance=vanilla-ewmtgg clusterdefinition.kubeblocks.io/name=vanilla-postgresql clusterversion.kubeblocks.io/name= Annotations: kubeblocks.io/ops-request: [***"name":"vanilla-ewmtgg-verticalscaling-tr9ft","type":"VerticalScaling"***] API Version: apps.kubeblocks.io/v1alpha1 Kind: Cluster Metadata: Creation Timestamp: 2025-09-01T03:17:21Z Finalizers: cluster.kubeblocks.io/finalizer Generation: 6 Resource Version: 22781 UID: afc63470-0298-4a0e-afa4-2743e92970fc Spec: Cluster Definition Ref: vanilla-postgresql Component Specs: Component Def: vanilla-postgresql-12 Name: postgresql Replicas: 2 Resources: Limits: Cpu: 600m Memory: 644245094400m Requests: Cpu: 600m Memory: 644245094400m Service Version: 12.15.0 Volume Claim Templates: Name: data Spec: Access Modes: ReadWriteOnce Resources: Requests: Storage: 3Gi Resources: Cpu: 0 Memory: 0 Services: Annotations: service.beta.kubernetes.io/azure-load-balancer-internal: true Component Selector: postgresql Name: postgresql-vpc Role Selector: primary Service Name: postgresql-vpc Spec: Ports: Name: tcp-postgresql Node Port: 30402 Port: 5432 Protocol: TCP Target Port: tcp-postgresql Type: LoadBalancer Storage: Size: 0 Termination Policy: Delete Topology: vanilla-postgresql Status: Cluster Def Generation: 1 Components: Postgresql: Phase: Updating Pods Ready: false Pods Ready Time: 2025-09-01T03:24:34Z Conditions: Last Transition Time: 2025-09-01T03:17:21Z Message: The operator has started the provisioning of Cluster: vanilla-ewmtgg Observed Generation: 6 Reason: PreCheckSucceed Status: True Type: ProvisioningStarted Last Transition Time: 2025-09-01T03:17:21Z Message: Successfully applied for resources Observed Generation: 6 Reason: ApplyResourcesSucceed Status: True Type: ApplyResources Last Transition Time: 2025-09-01T03:24:46Z Message: pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ReplicasNotReady Status: False Type: ReplicasReady Last Transition Time: 2025-09-01T03:24:46Z Message: pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Reason: ComponentsNotReady Status: False Type: Ready Observed Generation: 6 Phase: Updating Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal ComponentPhaseTransition 15m cluster-controller component is Creating Normal PreCheckSucceed 12m (x5 over 15m) cluster-controller The operator has started the provisioning of Cluster: vanilla-ewmtgg Normal ApplyResourcesSucceed 12m (x5 over 15m) cluster-controller Successfully applied for resources Warning ReplicasNotReady 10m cluster-controller pods are not ready in Components: [postgresql], refer to related component message in Cluster.status.components Warning ComponentsNotReady 10m cluster-controller pods are unavailable in Components: [postgresql], refer to related component message in Cluster.status.components Normal ComponentPhaseTransition 10m cluster-controller component is Abnormal Warning Abnormal 10m cluster-controller Cluster: vanilla-ewmtgg is Abnormal, check according to the components message Normal ComponentPhaseTransition 8m16s (x2 over 12m) cluster-controller component is Running Normal AllReplicasReady 8m16s (x2 over 12m) cluster-controller all pods of components are ready, waiting for the probe detection successful Normal ClusterReady 8m16s (x2 over 12m) cluster-controller Cluster: vanilla-ewmtgg is ready, current phase is Running Normal Running 8m16s (x2 over 12m) cluster-controller Cluster: vanilla-ewmtgg is ready, current phase is Running Normal ComponentPhaseTransition 8m15s (x3 over 10m) cluster-controller component is Updating ------------------------------------------------------------------------------------------------------------------ check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:24 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql PodInitializing 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 11:26 UTC+0800 pod_status:PodInitializing pod_status:PodInitializing check pod status done check cluster status again check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-verticalscaling-tr9ft ns-drhff VerticalScaling vanilla-ewmtgg postgresql Running 1/2 Sep 01,2025 11:24 UTC+0800 ops_status:vanilla-ewmtgg-verticalscaling-tr9ft ns-drhff VerticalScaling vanilla-ewmtgg postgresql Running 1/2 Sep 01,2025 11:24 UTC+0800 ops_status:vanilla-ewmtgg-verticalscaling-tr9ft ns-drhff VerticalScaling vanilla-ewmtgg postgresql Running 1/2 Sep 01,2025 11:24 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-verticalscaling-tr9ft ns-drhff VerticalScaling vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 11:24 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-verticalscaling-tr9ft --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-verticalscaling-tr9ft patched `kbcli cluster delete-ops --name vanilla-ewmtgg-verticalscaling-tr9ft --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-verticalscaling-tr9ft deleted test failover drainnode check cluster status before cluster-failover-drainnode check cluster status done cluster_status:Running check node drain check node drain success kubectl get pod vanilla-ewmtgg-postgresql-0 --namespace ns-drhff -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-68727890-vmss000006 success check if multiple pods are on the same node kubectl get pod vanilla-ewmtgg-postgresql-1 --namespace ns-drhff -o jsonpath='***.spec.nodeName***' get node name:aks-cicdamdpool-68727890-vmss000007 success kubectl drain aks-cicdamdpool-68727890-vmss000006 --delete-emptydir-data --ignore-daemonsets --force --grace-period 0 --timeout 60s node/aks-cicdamdpool-68727890-vmss000006 cordoned Warning: ignoring DaemonSet-managed Pods: chaos-mesh/chaos-daemon-ppjfv, kb-luwux/kb-addon-apecloud-otel-collector-jglbc, kube-system/azure-cns-9wczd, kube-system/azure-ip-masq-agent-pwvhd, kube-system/cloud-node-manager-69j8w, kube-system/csi-azuredisk-node-xslv5, kube-system/csi-azurefile-node-rl5r7, kube-system/kube-proxy-lczjv evicting pod ns-kxile/vastbase-zdplpq-vastbase-0 evicting pod kube-system/metrics-server-6b88d9f9c9-kbsnm evicting pod ns-fjjhy/weaviate-ntbdfz-weaviate-0 evicting pod ns-bwwyh/goldendb-fijseo-goldendb-cn-00-0 evicting pod ns-bwwyh/goldendb-fijseo-goldendb-gtm-0-0 evicting pod ns-drhff/vanilla-ewmtgg-postgresql-0 evicting pod ns-bwwyh/kb-post-provision-job-goldendb-fijseo-goldendb-dn-00-wmwpn evicting pod ns-hfqeb/mogdb-tzneyh-mogdb-0 evicting pod ns-istxe/px-gxg-dn-1 evicting pod ns-istxe/px-gxg-gms-1 evicting pod ns-bwwyh/goldendb-fijseo-goldendb-dn-00-0 pod/vastbase-zdplpq-vastbase-0 evicted pod/metrics-server-6b88d9f9c9-kbsnm evicted pod/mogdb-tzneyh-mogdb-0 evicted pod/goldendb-fijseo-goldendb-cn-00-0 evicted pod/weaviate-ntbdfz-weaviate-0 evicted pod/px-gxg-dn-1 evicted pod/goldendb-fijseo-goldendb-gtm-0-0 evicted pod/px-gxg-gms-1 evicted pod/goldendb-fijseo-goldendb-dn-00-0 evicted pod/vanilla-ewmtgg-postgresql-0 evicted pod/kb-post-provision-job-goldendb-fijseo-goldendb-dn-00-wmwpn evicted node/aks-cicdamdpool-68727890-vmss000006 drained kubectl uncordon aks-cicdamdpool-68727890-vmss000006 node/aks-cicdamdpool-68727890-vmss000006 uncordoned check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "vanilla-ewmtgg-postgresql-0" force deleted cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:36 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 11:26 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover drainnode Success cluster restart check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster restart vanilla-ewmtgg --auto-approve --force=true --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-restart-lskvv created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-restart-lskvv -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-restart-lskvv ns-drhff Restart vanilla-ewmtgg postgresql Running 0/2 Sep 01,2025 11:38 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:39 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-restart-lskvv ns-drhff Restart vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 11:38 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-restart-lskvv ns-drhff Restart vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 11:38 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-restart-lskvv --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-restart-lskvv patched `kbcli cluster delete-ops --name vanilla-ewmtgg-restart-lskvv --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-restart-lskvv deleted test failover oom check cluster status before cluster-failover-oom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-ewmtgg" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-oom-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all stressors: memory: workers: 1 size: "100GB" oomScoreAdj: -1000 duration: 2m `kubectl apply -f test-chaos-mesh-oom-vanilla-ewmtgg.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-oom-vanilla-ewmtgg created apply test-chaos-mesh-oom-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-oom-vanilla-ewmtgg.yaml` check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:39 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-oom-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-ewmtgg" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-oom-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover oom Success test failover networkdelay check cluster status before cluster-failover-networkdelay check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkdelay-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-1 mode: all action: delay delay: latency: 2000ms correlation: '100' jitter: 0ms direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkdelay-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkdelay-vanilla-ewmtgg created apply test-chaos-mesh-networkdelay-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkdelay-vanilla-ewmtgg.yaml` networkdelay chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:3Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:39 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkdelay-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkdelay-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover networkdelay Success `kubectl get pvc -l app.kubernetes.io/instance=vanilla-ewmtgg,apps.kubeblocks.io/component-name=postgresql,apps.kubeblocks.io/vct-name=data --namespace ns-drhff ` cluster volume-expand check cluster status before ops check cluster status done cluster_status:Running No resources found in vanilla-ewmtgg namespace. `kbcli cluster volume-expand vanilla-ewmtgg --auto-approve --force=true --components postgresql --volume-claim-templates data --storage 6Gi --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-volumeexpansion-88qlx created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-volumeexpansion-88qlx -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-volumeexpansion-88qlx ns-drhff VolumeExpansion vanilla-ewmtgg postgresql Running 0/2 Sep 01,2025 11:42 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:39 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done No resources found in vanilla-ewmtgg namespace. check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-volumeexpansion-88qlx ns-drhff VolumeExpansion vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 11:42 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-volumeexpansion-88qlx ns-drhff VolumeExpansion vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 11:42 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-volumeexpansion-88qlx --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-volumeexpansion-88qlx patched `kbcli cluster delete-ops --name vanilla-ewmtgg-volumeexpansion-88qlx --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-volumeexpansion-88qlx deleted test failover kill1 check cluster status before cluster-failover-kill1 check cluster status done cluster_status:Running check node drain check node drain success `kill 1` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) Unable to use a TTY - input is not a terminal or the right kind of file exec return message: check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:39 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 failover kill1 Success test failover networkcorruptover check cluster status before cluster-failover-networkcorruptover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkcorruptover-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all action: corrupt corrupt: corrupt: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkcorruptover-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkcorruptover-vanilla-ewmtgg created apply test-chaos-mesh-networkcorruptover-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkcorruptover-vanilla-ewmtgg.yaml` networkcorruptover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:39 UTC+0800 pod_status:Error pod_status:Error pod_status:Error pod_status:Error check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkcorruptover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkcorruptover-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover networkcorruptover Success test failover podkill check cluster status before cluster-failover-podkill check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-vanilla-ewmtgg" not found Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: PodChaos metadata: name: test-chaos-mesh-podkill-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-1 mode: all action: pod-kill `kubectl apply -f test-chaos-mesh-podkill-vanilla-ewmtgg.yaml` podchaos.chaos-mesh.org/test-chaos-mesh-podkill-vanilla-ewmtgg created apply test-chaos-mesh-podkill-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-podkill-vanilla-ewmtgg.yaml` check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge PodChaos test-chaos-mesh-podkill-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. podchaos.chaos-mesh.org "test-chaos-mesh-podkill-vanilla-ewmtgg" force deleted Error from server (NotFound): podchaos.chaos-mesh.org "test-chaos-mesh-podkill-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 failover podkill Success test failover dnserror check cluster status before cluster-failover-dnserror check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-ewmtgg" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnserror-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all action: error duration: 2m `kubectl apply -f test-chaos-mesh-dnserror-vanilla-ewmtgg.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnserror-vanilla-ewmtgg created apply test-chaos-mesh-dnserror-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-dnserror-vanilla-ewmtgg.yaml` dnserror chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnserror-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-ewmtgg" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnserror-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 failover dnserror Success 11 test failover networkduplicate check cluster status before cluster-failover-networkduplicate check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkduplicate-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all action: duplicate duplicate: duplicate: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkduplicate-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkduplicate-vanilla-ewmtgg created apply test-chaos-mesh-networkduplicate-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkduplicate-vanilla-ewmtgg.yaml` networkduplicate chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkduplicate-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkduplicate-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 failover networkduplicate Success test failover networkpartition check cluster status before cluster-failover-networkpartition check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkpartition-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 action: partition mode: all target: mode: all selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-1 direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networkpartition-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkpartition-vanilla-ewmtgg created apply test-chaos-mesh-networkpartition-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkpartition-vanilla-ewmtgg.yaml` networkpartition chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkpartition-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkpartition-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 failover networkpartition Success test failover networklossover check cluster status before cluster-failover-networklossover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networklossover-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all action: loss loss: loss: '100' correlation: '100' direction: to duration: 2m `kubectl apply -f test-chaos-mesh-networklossover-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networklossover-vanilla-ewmtgg created apply test-chaos-mesh-networklossover-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networklossover-vanilla-ewmtgg.yaml` networklossover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networklossover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networklossover-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover networklossover Success `kubectl get secrets -l app.kubernetes.io/instance=vanilla-ewmtgg` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:D4557qD3Na;DB_PORT:5432;DB_DATABASE: `create database benchtest;` Unable to use a TTY - input is not a terminal or the right kind of file NOTICE: database "benchtest" does not exist, skipping return msg:Connect to instance vanilla-ewmtgg-postgresql-1: out of vanilla-ewmtgg-postgresql-1(primary), vanilla-ewmtgg-postgresql-0(secondary) DROP DATABASE CREATE DATABASE apiVersion: v1 kind: Pod metadata: name: benchtest-vanilla-ewmtgg namespace: ns-drhff spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:pgsql,host:vanilla-ewmtgg-postgresql.ns-drhff.svc.cluster.local,user:postgres,password:D4557qD3Na,port:5432,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-vanilla-ewmtgg.yaml` pod/benchtest-vanilla-ewmtgg created apply benchtest-vanilla-ewmtgg.yaml Success check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 0/1 ContainerCreating 0 0s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 1/1 Running 0 10s check pod benchtest-vanilla-ewmtgg status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 0/1 Completed 0 16s `rm -rf benchtest-vanilla-ewmtgg.yaml` `kubectl logs benchtest-vanilla-ewmtgg --tail 30 --namespace ns-drhff ` [ 7s ] thds: 4 tps: 346.01 qps: 6932.15 (r/w/o: 4854.11/1383.03/695.02) lat (ms,99%): 995.51 err/s: 1.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 438.62 qps: 8818.26 (r/w/o: 6178.58/1751.46/888.22) lat (ms,99%): 82.96 err/s: 3.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 457.41 qps: 9200.32 (r/w/o: 6438.83/1838.66/922.83) lat (ms,99%): 81.48 err/s: 4.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 401.78 qps: 8051.61 (r/w/o: 5640.93/1602.13/808.56) lat (ms,99%): 82.96 err/s: 0.00 reconn/s: 0.00 SQL statistics: queries performed: read: 45010 write: 12805 other: 6451 total: 64266 transactions: 3200 (318.02 per sec.) queries: 64266 (6386.92 per sec.) ignored errors: 15 (1.49 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0601s total number of events: 3200 Latency (ms): min: 1.44 avg: 12.57 max: 1081.57 99th percentile: 82.96 sum: 40229.17 Threads fairness: events (avg/stddev): 800.0000/17.76 execution time (avg/stddev): 10.0573/0.00 `kubectl delete pod benchtest-vanilla-ewmtgg --force --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-vanilla-ewmtgg" force deleted No resources found in ns-drhff namespace. `kubectl get secrets -l app.kubernetes.io/instance=vanilla-ewmtgg` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.username***"` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.password***"` `kubectl get secrets vanilla-ewmtgg-postgresql-account-postgres -o jsonpath="***.data.port***"` DB_USERNAME:postgres;DB_PASSWORD:D4557qD3Na;DB_PORT:5432;DB_DATABASE: `create database benchtest;` Unable to use a TTY - input is not a terminal or the right kind of file return msg:Connect to instance vanilla-ewmtgg-postgresql-1: out of vanilla-ewmtgg-postgresql-1(primary), vanilla-ewmtgg-postgresql-0(secondary) DROP DATABASE CREATE DATABASE apiVersion: v1 kind: Pod metadata: name: benchtest-vanilla-ewmtgg namespace: ns-drhff spec: containers: - name: test-sysbench imagePullPolicy: IfNotPresent image: docker.io/apecloud/customsuites:latest env: - name: TYPE value: "2" - name: FLAG value: "0" - name: CONFIGS value: "mode:all,driver:pgsql,host:10.224.0.6,user:postgres,password:D4557qD3Na,port:5432,db:benchtest,tables:5,threads:4,times:10,size:1000,type:oltp_read_write" restartPolicy: Never `kubectl apply -f benchtest-vanilla-ewmtgg.yaml` pod/benchtest-vanilla-ewmtgg created apply benchtest-vanilla-ewmtgg.yaml Success check pod status pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 0/1 ContainerCreating 0 1s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 1/1 Running 0 5s pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 1/1 Running 0 11s check pod benchtest-vanilla-ewmtgg status done pod_status:NAME READY STATUS RESTARTS AGE benchtest-vanilla-ewmtgg 0/1 Completed 0 16s `rm -rf benchtest-vanilla-ewmtgg.yaml` `kubectl logs benchtest-vanilla-ewmtgg --tail 30 --namespace ns-drhff ` [ 7s ] thds: 4 tps: 367.02 qps: 7381.34 (r/w/o: 5171.24/1467.07/743.03) lat (ms,99%): 86.00 err/s: 3.00 reconn/s: 0.00 [ 8s ] thds: 4 tps: 448.02 qps: 8991.38 (r/w/o: 6296.27/1790.08/905.04) lat (ms,99%): 81.48 err/s: 2.00 reconn/s: 0.00 [ 9s ] thds: 4 tps: 203.00 qps: 4079.09 (r/w/o: 2855.06/817.02/407.01) lat (ms,99%): 81.48 err/s: 0.00 reconn/s: 0.00 [ 10s ] thds: 4 tps: 261.97 qps: 5246.45 (r/w/o: 3676.61/1040.89/528.94) lat (ms,99%): 995.51 err/s: 1.00 reconn/s: 0.00 SQL statistics: queries performed: read: 48272 write: 13742 other: 6921 total: 68935 transactions: 3437 (342.51 per sec.) queries: 68935 (6869.73 per sec.) ignored errors: 11 (1.10 per sec.) reconnects: 0 (0.00 per sec.) General statistics: total time: 10.0325s total number of events: 3437 Latency (ms): min: 1.44 avg: 11.67 max: 1005.11 99th percentile: 84.47 sum: 40118.07 Threads fairness: events (avg/stddev): 859.2500/31.64 execution time (avg/stddev): 10.0295/0.00 `kubectl delete pod benchtest-vanilla-ewmtgg --force --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "benchtest-vanilla-ewmtgg" force deleted test failover fullcpu check cluster status before cluster-failover-fullcpu check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-ewmtgg" not found Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: StressChaos metadata: name: test-chaos-mesh-fullcpu-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-1 mode: all stressors: cpu: workers: 100 load: 100 duration: 2m `kubectl apply -f test-chaos-mesh-fullcpu-vanilla-ewmtgg.yaml` stresschaos.chaos-mesh.org/test-chaos-mesh-fullcpu-vanilla-ewmtgg created apply test-chaos-mesh-fullcpu-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-fullcpu-vanilla-ewmtgg.yaml` fullcpu chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge StressChaos test-chaos-mesh-fullcpu-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-ewmtgg" force deleted Error from server (NotFound): stresschaos.chaos-mesh.org "test-chaos-mesh-fullcpu-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover fullcpu Success cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in vanilla-ewmtgg namespace. `kbcli cluster hscale vanilla-ewmtgg --auto-approve --force=true --components postgresql --replicas 3 --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-horizontalscaling-fgdq5 created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-horizontalscaling-fgdq5 -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-horizontalscaling-fgdq5 ns-drhff HorizontalScaling vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:06 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 vanilla-ewmtgg-postgresql-2 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 12:06 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 vanilla-ewmtgg-postgresql-2 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done No resources found in vanilla-ewmtgg namespace. check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-horizontalscaling-fgdq5 ns-drhff HorizontalScaling vanilla-ewmtgg postgresql Succeed 1/1 Sep 01,2025 12:06 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-horizontalscaling-fgdq5 ns-drhff HorizontalScaling vanilla-ewmtgg postgresql Succeed 1/1 Sep 01,2025 12:06 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-horizontalscaling-fgdq5 --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-horizontalscaling-fgdq5 patched `kbcli cluster delete-ops --name vanilla-ewmtgg-horizontalscaling-fgdq5 --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-horizontalscaling-fgdq5 deleted cluster hscale check cluster status before ops check cluster status done cluster_status:Running No resources found in vanilla-ewmtgg namespace. `kbcli cluster hscale vanilla-ewmtgg --auto-approve --force=true --components postgresql --replicas 2 --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-horizontalscaling-65hbd created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-horizontalscaling-65hbd -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-horizontalscaling-65hbd ns-drhff HorizontalScaling vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:07 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 11:38 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 11:50 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done No resources found in vanilla-ewmtgg namespace. check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-horizontalscaling-65hbd ns-drhff HorizontalScaling vanilla-ewmtgg postgresql Succeed 1/1 Sep 01,2025 12:07 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-horizontalscaling-65hbd ns-drhff HorizontalScaling vanilla-ewmtgg postgresql Succeed 1/1 Sep 01,2025 12:07 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-horizontalscaling-65hbd --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-horizontalscaling-65hbd patched `kbcli cluster delete-ops --name vanilla-ewmtgg-horizontalscaling-65hbd --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-horizontalscaling-65hbd deleted cluster stop check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster stop vanilla-ewmtgg --auto-approve --force=true --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-stop-84xpv created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-stop-84xpv -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-stop-84xpv ns-drhff Stop vanilla-ewmtgg postgresql Running 0/2 Sep 01,2025 12:08 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Stopped Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Stopped check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME check pod status done check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-stop-84xpv ns-drhff Stop vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 12:08 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-stop-84xpv ns-drhff Stop vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 12:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-stop-84xpv --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-stop-84xpv patched `kbcli cluster delete-ops --name vanilla-ewmtgg-stop-84xpv --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-stop-84xpv deleted cluster start check cluster status before ops check cluster status done cluster_status:Stopped `kbcli cluster start vanilla-ewmtgg --force=true --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-start-glkbx created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-start-glkbx -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-start-glkbx ns-drhff Start vanilla-ewmtgg postgresql Running 0/2 Sep 01,2025 12:08 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 12:08 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-start-glkbx ns-drhff Start vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 12:08 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-start-glkbx ns-drhff Start vanilla-ewmtgg postgresql Succeed 2/2 Sep 01,2025 12:08 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-start-glkbx --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-start-glkbx patched `kbcli cluster delete-ops --name vanilla-ewmtgg-start-glkbx --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-start-glkbx deleted skip Reconfiguring: test failover dnsrandom check cluster status before cluster-failover-dnsrandom check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-ewmtgg" not found Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: DNSChaos metadata: name: test-chaos-mesh-dnsrandom-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 mode: all action: random duration: 2m `kubectl apply -f test-chaos-mesh-dnsrandom-vanilla-ewmtgg.yaml` dnschaos.chaos-mesh.org/test-chaos-mesh-dnsrandom-vanilla-ewmtgg created apply test-chaos-mesh-dnsrandom-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-dnsrandom-vanilla-ewmtgg.yaml` dnsrandom chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 12:08 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge DNSChaos test-chaos-mesh-dnsrandom-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-ewmtgg" force deleted Error from server (NotFound): dnschaos.chaos-mesh.org "test-chaos-mesh-dnsrandom-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 failover dnsrandom Success skip Reconfiguring: test failover networkbandwidthover check cluster status before cluster-failover-networkbandwidthover check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg created apply test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000006/10.224.0.10 Sep 01,2025 12:08 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg created apply test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Failed Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal cluster_status:Abnormal check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 12:14 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-0;secondary: vanilla-ewmtgg-postgresql-1 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found failover pod name:vanilla-ewmtgg-postgresql-0 checking failover... `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: NetworkChaos metadata: name: test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-0 action: bandwidth mode: all bandwidth: rate: '1bps' limit: 20971520 buffer: 10000 duration: 2m `kubectl apply -f test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml` networkchaos.chaos-mesh.org/test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg created apply test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg.yaml` networkbandwidthover chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Updating Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= cluster_status:Updating cluster_status:Updating cluster_status:Updating cluster_status:Updating check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 12:14 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge NetworkChaos test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" force deleted Error from server (NotFound): networkchaos.chaos-mesh.org "test-chaos-mesh-networkbandwidthover-vanilla-ewmtgg" not found check failover pod name:vanilla-ewmtgg-postgresql-1 failover networkbandwidthover Success test failover timeoffset check cluster status before cluster-failover-timeoffset check cluster status done cluster_status:Running check node drain check node drain success `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-ewmtgg" not found Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-ewmtgg" not found apiVersion: chaos-mesh.org/v1alpha1 kind: TimeChaos metadata: name: test-chaos-mesh-timeoffset-vanilla-ewmtgg namespace: ns-drhff spec: selector: namespaces: - ns-drhff labelSelectors: apps.kubeblocks.io/pod-name: vanilla-ewmtgg-postgresql-1 mode: all timeOffset: '-10m' clockIds: - CLOCK_REALTIME duration: 2m `kubectl apply -f test-chaos-mesh-timeoffset-vanilla-ewmtgg.yaml` timechaos.chaos-mesh.org/test-chaos-mesh-timeoffset-vanilla-ewmtgg created apply test-chaos-mesh-timeoffset-vanilla-ewmtgg.yaml Success `rm -rf test-chaos-mesh-timeoffset-vanilla-ewmtgg.yaml` timeoffset chaos test waiting 120 seconds check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql Delete Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 12:14 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge TimeChaos test-chaos-mesh-timeoffset-vanilla-ewmtgg --namespace ns-drhff ` Warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-ewmtgg" force deleted Error from server (NotFound): timechaos.chaos-mesh.org "test-chaos-mesh-timeoffset-vanilla-ewmtgg" not found check failover pod name failover pod name:vanilla-ewmtgg-postgresql-1 failover timeoffset Success cluster update terminationPolicy WipeOut `kbcli cluster update vanilla-ewmtgg --termination-policy=WipeOut --namespace ns-drhff ` cluster.apps.kubeblocks.io/vanilla-ewmtgg updated check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql WipeOut Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 12:14 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done cluster list-logs `kbcli cluster list-logs vanilla-ewmtgg --namespace ns-drhff ` No log files found. You can enable the log feature with the kbcli command below. kbcli cluster update vanilla-ewmtgg --enable-all-logs=true --namespace ns-drhff Error from server (NotFound): pods "vanilla-ewmtgg-postgresql-1" not found cluster logs `kbcli cluster logs vanilla-ewmtgg --tail 30 --namespace ns-drhff ` Defaulted container "postgresql" out of: postgresql, lorry, config-manager, init-syncer (init), install-config-manager-tool (init) == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:04.615 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:05.600 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:06.613 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:07.611 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:08.602 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:09.601 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == == DB ERR == 2025-09-01 04:22:10.605 GMT [59] LOG: statement: == DB ERR == create table if not exists kb_health_check(type int, check_ts timestamp, primary key(type)); == DB ERR == insert into kb_health_check values(1, CURRENT_TIMESTAMP) on conflict(type) do update set check_ts = CURRENT_TIMESTAMP; == DB ERR == cluster expose check cluster status before ops check cluster status done cluster_status:Running `kbcli cluster expose vanilla-ewmtgg --auto-approve --force=true --type vpc --enable false --components postgresql --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-expose-f5z6t created successfully, you can view the progress: kbcli cluster describe-ops vanilla-ewmtgg-expose-f5z6t -n ns-drhff check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 check cluster status `kbcli cluster list vanilla-ewmtgg --show-labels --namespace ns-drhff ` NAME NAMESPACE CLUSTER-DEFINITION VERSION TERMINATION-POLICY STATUS CREATED-TIME LABELS vanilla-ewmtgg ns-drhff vanilla-postgresql WipeOut Running Sep 01,2025 11:17 UTC+0800 app.kubernetes.io/instance=vanilla-ewmtgg,clusterdefinition.kubeblocks.io/name=vanilla-postgresql,clusterversion.kubeblocks.io/name= check cluster status done cluster_status:Running check pod status `kbcli cluster list-instances vanilla-ewmtgg --namespace ns-drhff ` NAME NAMESPACE CLUSTER COMPONENT STATUS ROLE ACCESSMODE AZ CPU(REQUEST/LIMIT) MEMORY(REQUEST/LIMIT) STORAGE NODE CREATED-TIME vanilla-ewmtgg-postgresql-0 ns-drhff vanilla-ewmtgg postgresql Running secondary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000008/10.224.0.9 Sep 01,2025 12:08 UTC+0800 vanilla-ewmtgg-postgresql-1 ns-drhff vanilla-ewmtgg postgresql Running primary 0 600m / 600m 644245094400m / 644245094400m data:6Gi aks-cicdamdpool-68727890-vmss000007/10.224.0.5 Sep 01,2025 12:14 UTC+0800 check pod status done check cluster role check cluster role done primary: vanilla-ewmtgg-postgresql-1;secondary: vanilla-ewmtgg-postgresql-0 check cluster connect `echo '' | kbcli cluster connect vanilla-ewmtgg --namespace ns-drhff` check cluster connect done check ops status `kbcli cluster list-ops vanilla-ewmtgg --status all --namespace ns-drhff ` NAME NAMESPACE TYPE CLUSTER COMPONENT STATUS PROGRESS CREATED-TIME vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Running 0/1 Sep 01,2025 12:22 UTC+0800 check ops status done ops_status:vanilla-ewmtgg-expose-f5z6t ns-drhff Expose vanilla-ewmtgg postgresql Succeed 1/1 Sep 01,2025 12:22 UTC+0800 `kubectl patch -p '***"metadata":***"finalizers":[]***' --type=merge opsrequests vanilla-ewmtgg-expose-f5z6t --namespace ns-drhff ` opsrequest.apps.kubeblocks.io/vanilla-ewmtgg-expose-f5z6t patched `kbcli cluster delete-ops --name vanilla-ewmtgg-expose-f5z6t --force --auto-approve --namespace ns-drhff ` OpsRequest vanilla-ewmtgg-expose-f5z6t deleted delete cluster vanilla-ewmtgg `kbcli cluster delete vanilla-ewmtgg --auto-approve --namespace ns-drhff ` Cluster vanilla-ewmtgg deleted pod_info:vanilla-ewmtgg-postgresql-0 3/3 Running 2 (3m53s ago) 14m vanilla-ewmtgg-postgresql-1 3/3 Running 4 (6m50s ago) 8m29s No resources found in ns-drhff namespace. delete cluster pod done No resources found in ns-drhff namespace. check cluster resource non-exist OK: pvc No resources found in ns-drhff namespace. delete cluster done No resources found in ns-drhff namespace. No resources found in ns-drhff namespace. No resources found in ns-drhff namespace. Vanilla Postgresql Test Suite All Done! --------------------------------------Vanilla Postgresql (Topology = vanilla-postgresql Replicas 2) Test Result-------------------------------------- [PASSED]|[Create]|[Topology=vanilla-postgresql;ComponentVersion=vanilla-postgresql;ServiceVersion=12.15.0;]|[Description=Create a cluster with the specified topology vanilla-postgresql and component version vanilla-postgresql and service version 12.15.0] [PASSED]|[Connect]|[ComponentName=postgresql]|[Description=Connect to the cluster] [PASSED]|[Expose]|[Enable=true;TYPE=vpc;ComponentName=postgresql]|[Description=Expose Enable the vpc service with postgresql component] [PASSED]|[Failover]|[HA=Pod Failure;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience failure for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to failure.] [PASSED]|[VerticalScaling]|[ComponentName=postgresql]|[Description=VerticalScaling the cluster specify component postgresql] [PASSED]|[Failover]|[HA=Evicting Pod;ComponentName=postgresql]|[Description=Simulates conditions where pods evicting either due to node drained thereby testing the application's resilience to unavailability of some replicas due to evicting.] [PASSED]|[Restart]|[-]|[Description=Restart the cluster] [PASSED]|[Failover]|[HA=OOM;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience OOM either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high Memory load.] [PASSED]|[No-Failover]|[HA=Network Delay;Durations=2m;ComponentName=postgresql]|[Description=Simulates network delay fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to delay network.] [PASSED]|[VolumeExpansion]|[ComponentName=postgresql]|[Description=VolumeExpansion the cluster specify component postgresql] [PASSED]|[Failover]|[HA=Kill 1;ComponentName=postgresql]|[Description=Simulates conditions where process 1 killed either due to expected/undesired processes thereby testing the application's resilience to unavailability of some replicas due to abnormal termination signals.] [PASSED]|[Failover]|[HA=Network Corrupt;Durations=2m;ComponentName=postgresql]|[Description=Simulates network corrupt fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to corrupt network.] [PASSED]|[Failover]|[HA=Pod Kill;ComponentName=postgresql]|[Description=Simulates conditions where pods experience kill for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to kill.] [PASSED]|[No-Failover]|[HA=DNS Error;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience DNS service errors for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to DNS service errors.] [PASSED]|[No-Failover]|[HA=Network Duplicate;Durations=2m;ComponentName=postgresql]|[Description=Simulates network duplicate fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to duplicate network.] [PASSED]|[No-Failover]|[HA=Network Partition;Durations=2m;ComponentName=postgresql]|[Description=Simulates network partition fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to partition network.] [PASSED]|[Failover]|[HA=Network Loss;Durations=2m;ComponentName=postgresql]|[Description=Simulates network loss fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to loss network.] [PASSED]|[Bench]|[ComponentName=postgresql]|[Description=Bench the cluster service with postgresql component] [PASSED]|[Bench]|[HostType=LB;ComponentName=postgresql]|[Description=Bench the cluster LB service with postgresql component] [PASSED]|[No-Failover]|[HA=Full CPU;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience CPU full either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to high CPU load.] [PASSED]|[HorizontalScaling Out]|[ComponentName=postgresql]|[Description=HorizontalScaling Out the cluster specify component postgresql] [PASSED]|[HorizontalScaling In]|[ComponentName=postgresql]|[Description=HorizontalScaling In the cluster specify component postgresql] [PASSED]|[Stop]|[-]|[Description=Stop the cluster] [PASSED]|[Start]|[-]|[Description=Start the cluster] [PASSED]|[No-Failover]|[HA=DNS Random;Durations=2m;ComponentName=postgresql]|[Description=Simulates conditions where pods experience random IP addresses being returned by the DNS service for a period of time either due to expected/undesired processes thereby testing the application's resilience to potential slowness/unavailability of some replicas due to the DNS service returning random IP addresses.] [PASSED]|[Failover]|[HA=Network Bandwidth;Durations=2m;ComponentName=postgresql]|[Description=Simulates network bandwidth fault thereby testing the application's resilience to potential slowness/unavailability of some replicas due to bandwidth network.] [PASSED]|[No-Failover]|[HA=Time Offset;Durations=2m;ComponentName=postgresql]|[Description=Simulates a time offset scenario thereby testing the application's resilience to potential slowness/unavailability of some replicas due to time offset.] [PASSED]|[Update]|[TerminationPolicy=WipeOut]|[Description=Update the cluster TerminationPolicy WipeOut] [PASSED]|[Expose]|[Disable=true;TYPE=vpc;ComponentName=postgresql]|[Description=Expose Disable the vpc service with postgresql component] [PASSED]|[Delete]|[-]|[Description=Delete the cluster] [END]